Table of contents
  1. Story
    1. Five Data Points You Need to Know for Farming
    2. Big Data Science for Precision Farming Business
    3. Smarter Farming with USDA Data
    4. Slides
      1. Slide 1 USDA: Open Data, Big Data and Precision Ag
      2. Slide 2 USDA Open Data 1
      3. Slide 3 USDA Open Data 2
      4. Slide 4 Open Data Projects
      5. Slide 5 Intersection of Open Data, Big Data and Open Government
      6. Slide 6 What is Precision Ag?
      7. Slide 7 Precision Ag and Big Data
      8. Slide 8 What Is Generating Big Data?
      9. Slide 9 Value of Precision Ag Data
      10. Slide 10 Decision Support for Farmers
      11. Slide 11 Farmer Profitability Strategy
      12. Slide 12 Precision Ag Delivers…
      13. Slide 13 Smarter Farming with USDA Data
      14. Slide 14 Questions
  2. Story
    1. Data Science Audit Results
    2. Script for Slide YouTube Demo Video 1
    3. Script for Live YouTube Demo Video 2
      1. Background
      2. Script
      3. Data-Driven Farming, January 25 - March 18, 2016
    4. Slides
      1. Slide 1 Farm Data Dashboards: USDA and Microsoft Innovation Challenge
      2. Slide 2 Dr. Brand L. Niemann
      3. Slide 3 USDA Data Science MOOC: AMS Geo US Farmers Markets Spotfire
      4. Slide 4 Teaching Farmers to Work with Big Data
      5. Slide 5 Week 7: Project Logs
      6. Slide 6 USDA, Microsoft to Launch “Innovation Challenge”
      7. Slide 7 Requirements
      8. Slide 8 Results
      9. Slide 9 USDA Farm Data Dashboard
      10. Slide 10 Spotfire Farm Data Dashboard
  3. Spotfire Dashboard
  4. Story
    1. Big Data Science for Precision Farming Business
    2. Slides and Videos
    3. Introduction
      1. Slide 1
      2. Slide 2
      3. Slide 3
    4. Week 1: Business Understanding
      1. Slide 4
      2. Slide 5
      3. Slide 6
      4. Slide 7
      5. Slide 8
      6. Slide 9
      7. Slide 10
    5. Week 2: Data Understanding
      1. Slide 11
      2. Slide 12
      3. Slide 13
      4. Slide 14
      5. Slide 15
      6. Slide 16
      7. Slide 17
      8. Slide 18
      9. Slide 19
      10. Slide 20
    6. Week 3: Data Preparation
      1. Slide 21
      2. Slide 22
      3. Slide 23
      4. Slide 24
      5. Slide 25
      6. Slide 26
      7. Slide 27
      8. Slide 28
      9. Slide 29
      10. Slide 30
      11. Slide 31
      12. Slide 32
      13. Slide 33
      14. Slide 34
    7. Week 4: Modeling
      1. Slide 35
      2. Slide 36
      3. Slide 37
      4. Slide 38
      5. Slide 39
      6. Slide 40
      7. Slide 41
      8. Slide 42
      9. Slide 43
      10. Slide 44
      11. Slide 45
      12. Slide 46
      13. Slide 47
    8. Week 5: Evaluation
      1. Slide 48
      2. Slide 49
      3. Slide 50
      4. Slide 51
      5. Slide 52
      6. Slide 53
      7. Slide 54
      8. Slide 55
      9. Slide 56
      10. Slide 57
      11. Slide 58
      12. Slide 59
      13. Slide 60
      14. Slide 61
      15. Slide 62
      16. Slide 63
      17. Slide 64
    9. Week 6: Deployment
      1. Slide 65
      2. Slide 66
      3. Slide 67
      4. Slide 68
      5. Slide 69
      6. Slide 70
      7. Slide 71
      8. Slide 72
      9. Slide 73
      10. Slide 74
      11. Slide 75
      12. Slide 76
      13. Slide 77
      14. Slide 78
      15. Slide 79
      16. Slide 80
      17. Slide 81
      18. Slide 82
      19. Slide 83
      20. Slide 84
      21. Slide 85
      22. Slide 86
    10. Week 7: Project
      1. Slide 87
      2. Slide 88
      3. Slide 89
      4. Slide 90
      5. Slide 91
      6. Slide 92
      7. Slide 93
    11. Week 8: Final Exam
      1. Slide 94
  5. Spotfire Dashboard
  6. USDA, Microsoft to Launch “Innovation Challenge”
  7. USDA and Microsoft Innovation Challenge for Food Resilience
    1. YouTube Video
    2. Transcript
    3. Help build a sustainable US food system by putting USDA data into the hands of farmers, researchers, and consumers
      1. Eligibility
      2. Requirements
      3. Judges
        1. Dr. Debra Peters
        2. Vince Breneman
        3. Rick Mueller
        4. Lucas Joppa
        5. Daron Green
        6. Dr. Brian Lutz
      4. Judging Criteria
      5. View All Dates
      6. Prizes
      7. How To Enter
    4. Beginning Farmer and Rancher Opportunities
    5. Technology is king down on the farm
    6. Participants
    7. Rules
    8. Submissions
    9. Updates
    10. Discussions
    11. Resources & Data
      1. Required USDA Datasets
      2. To easily access the data via Azure, visit the Farm Data Dashboard 
        1. NASS (Quickstats)
        2. ARMS
        3. CropScape
        4. VegScape
        5. Use cases
        6. AgMIP
        7. Documentation
        8. Download
      3. Support & Documentation
      4. Additional Resources
      5. Azure Resources
      6. Getting Started with an Idea
      7. Requirements for Bonus Categories
        1. Best Student-made App Award
        2. Best Visualization in Time or Space Award
        3. Best Open Source Application Award
      8. More Questions?
    12. FAQ
  8. Story
  9. Slides
    1. Slide 1 USDA Big Data Science for Precision Farming With FarmLogs
    2. Slide 2
    3. Slide 3
    4. Slide 4
    5. Slide 5
    6. Slide 6
    7. Slide 7
    8. Slide 8
    9. Slide 9
    10. Slide 10
    11. Slide 11
    12. Slide 12
    13. Slide 13
    14. Slide 14
    15. Slide 15
    16. Slide 16
  10. Research Notes
    1. Upload Shapefiles
    2. Google: fairfax county gis shapefiles for agriculture
    3. Data and Statistics
    4. Future Vision for the Use of Remotely Sensed Data
    5. Organic Farming Survey
  11. FarmLogs App Helps Farmers Collect and Retrieve Data
    1. Bushels and Bytes: The Data-Driven Farm
    2. Farmers Flying Drones May Soon Be Given Clearance
    3. Working the Land and the Data
    4. Correction: June 24, 2015 
    5. Note: June 25, 2015
  12. Google didn’t lead the self-driving vehicle revolution
  13. Drought Is the Mother of Invention on the Farm
  14. NEXT

Big Data Science for Precision Farming Business

Last modified
Table of contents
  1. Story
    1. Five Data Points You Need to Know for Farming
    2. Big Data Science for Precision Farming Business
    3. Smarter Farming with USDA Data
    4. Slides
      1. Slide 1 USDA: Open Data, Big Data and Precision Ag
      2. Slide 2 USDA Open Data 1
      3. Slide 3 USDA Open Data 2
      4. Slide 4 Open Data Projects
      5. Slide 5 Intersection of Open Data, Big Data and Open Government
      6. Slide 6 What is Precision Ag?
      7. Slide 7 Precision Ag and Big Data
      8. Slide 8 What Is Generating Big Data?
      9. Slide 9 Value of Precision Ag Data
      10. Slide 10 Decision Support for Farmers
      11. Slide 11 Farmer Profitability Strategy
      12. Slide 12 Precision Ag Delivers…
      13. Slide 13 Smarter Farming with USDA Data
      14. Slide 14 Questions
  2. Story
    1. Data Science Audit Results
    2. Script for Slide YouTube Demo Video 1
    3. Script for Live YouTube Demo Video 2
      1. Background
      2. Script
      3. Data-Driven Farming, January 25 - March 18, 2016
    4. Slides
      1. Slide 1 Farm Data Dashboards: USDA and Microsoft Innovation Challenge
      2. Slide 2 Dr. Brand L. Niemann
      3. Slide 3 USDA Data Science MOOC: AMS Geo US Farmers Markets Spotfire
      4. Slide 4 Teaching Farmers to Work with Big Data
      5. Slide 5 Week 7: Project Logs
      6. Slide 6 USDA, Microsoft to Launch “Innovation Challenge”
      7. Slide 7 Requirements
      8. Slide 8 Results
      9. Slide 9 USDA Farm Data Dashboard
      10. Slide 10 Spotfire Farm Data Dashboard
  3. Spotfire Dashboard
  4. Story
    1. Big Data Science for Precision Farming Business
    2. Slides and Videos
    3. Introduction
      1. Slide 1
      2. Slide 2
      3. Slide 3
    4. Week 1: Business Understanding
      1. Slide 4
      2. Slide 5
      3. Slide 6
      4. Slide 7
      5. Slide 8
      6. Slide 9
      7. Slide 10
    5. Week 2: Data Understanding
      1. Slide 11
      2. Slide 12
      3. Slide 13
      4. Slide 14
      5. Slide 15
      6. Slide 16
      7. Slide 17
      8. Slide 18
      9. Slide 19
      10. Slide 20
    6. Week 3: Data Preparation
      1. Slide 21
      2. Slide 22
      3. Slide 23
      4. Slide 24
      5. Slide 25
      6. Slide 26
      7. Slide 27
      8. Slide 28
      9. Slide 29
      10. Slide 30
      11. Slide 31
      12. Slide 32
      13. Slide 33
      14. Slide 34
    7. Week 4: Modeling
      1. Slide 35
      2. Slide 36
      3. Slide 37
      4. Slide 38
      5. Slide 39
      6. Slide 40
      7. Slide 41
      8. Slide 42
      9. Slide 43
      10. Slide 44
      11. Slide 45
      12. Slide 46
      13. Slide 47
    8. Week 5: Evaluation
      1. Slide 48
      2. Slide 49
      3. Slide 50
      4. Slide 51
      5. Slide 52
      6. Slide 53
      7. Slide 54
      8. Slide 55
      9. Slide 56
      10. Slide 57
      11. Slide 58
      12. Slide 59
      13. Slide 60
      14. Slide 61
      15. Slide 62
      16. Slide 63
      17. Slide 64
    9. Week 6: Deployment
      1. Slide 65
      2. Slide 66
      3. Slide 67
      4. Slide 68
      5. Slide 69
      6. Slide 70
      7. Slide 71
      8. Slide 72
      9. Slide 73
      10. Slide 74
      11. Slide 75
      12. Slide 76
      13. Slide 77
      14. Slide 78
      15. Slide 79
      16. Slide 80
      17. Slide 81
      18. Slide 82
      19. Slide 83
      20. Slide 84
      21. Slide 85
      22. Slide 86
    10. Week 7: Project
      1. Slide 87
      2. Slide 88
      3. Slide 89
      4. Slide 90
      5. Slide 91
      6. Slide 92
      7. Slide 93
    11. Week 8: Final Exam
      1. Slide 94
  5. Spotfire Dashboard
  6. USDA, Microsoft to Launch “Innovation Challenge”
  7. USDA and Microsoft Innovation Challenge for Food Resilience
    1. YouTube Video
    2. Transcript
    3. Help build a sustainable US food system by putting USDA data into the hands of farmers, researchers, and consumers
      1. Eligibility
      2. Requirements
      3. Judges
        1. Dr. Debra Peters
        2. Vince Breneman
        3. Rick Mueller
        4. Lucas Joppa
        5. Daron Green
        6. Dr. Brian Lutz
      4. Judging Criteria
      5. View All Dates
      6. Prizes
      7. How To Enter
    4. Beginning Farmer and Rancher Opportunities
    5. Technology is king down on the farm
    6. Participants
    7. Rules
    8. Submissions
    9. Updates
    10. Discussions
    11. Resources & Data
      1. Required USDA Datasets
      2. To easily access the data via Azure, visit the Farm Data Dashboard 
        1. NASS (Quickstats)
        2. ARMS
        3. CropScape
        4. VegScape
        5. Use cases
        6. AgMIP
        7. Documentation
        8. Download
      3. Support & Documentation
      4. Additional Resources
      5. Azure Resources
      6. Getting Started with an Idea
      7. Requirements for Bonus Categories
        1. Best Student-made App Award
        2. Best Visualization in Time or Space Award
        3. Best Open Source Application Award
      8. More Questions?
    12. FAQ
  8. Story
  9. Slides
    1. Slide 1 USDA Big Data Science for Precision Farming With FarmLogs
    2. Slide 2
    3. Slide 3
    4. Slide 4
    5. Slide 5
    6. Slide 6
    7. Slide 7
    8. Slide 8
    9. Slide 9
    10. Slide 10
    11. Slide 11
    12. Slide 12
    13. Slide 13
    14. Slide 14
    15. Slide 15
    16. Slide 16
  10. Research Notes
    1. Upload Shapefiles
    2. Google: fairfax county gis shapefiles for agriculture
    3. Data and Statistics
    4. Future Vision for the Use of Remotely Sensed Data
    5. Organic Farming Survey
  11. FarmLogs App Helps Farmers Collect and Retrieve Data
    1. Bushels and Bytes: The Data-Driven Farm
    2. Farmers Flying Drones May Soon Be Given Clearance
    3. Working the Land and the Data
    4. Correction: June 24, 2015 
    5. Note: June 25, 2015
  12. Google didn’t lead the self-driving vehicle revolution
  13. Drought Is the Mother of Invention on the Farm
  14. NEXT

  1. Story
    1. Five Data Points You Need to Know for Farming
    2. Big Data Science for Precision Farming Business
    3. Smarter Farming with USDA Data
    4. Slides
      1. Slide 1 USDA: Open Data, Big Data and Precision Ag
      2. Slide 2 USDA Open Data 1
      3. Slide 3 USDA Open Data 2
      4. Slide 4 Open Data Projects
      5. Slide 5 Intersection of Open Data, Big Data and Open Government
      6. Slide 6 What is Precision Ag?
      7. Slide 7 Precision Ag and Big Data
      8. Slide 8 What Is Generating Big Data?
      9. Slide 9 Value of Precision Ag Data
      10. Slide 10 Decision Support for Farmers
      11. Slide 11 Farmer Profitability Strategy
      12. Slide 12 Precision Ag Delivers…
      13. Slide 13 Smarter Farming with USDA Data
      14. Slide 14 Questions
  2. Story
    1. Data Science Audit Results
    2. Script for Slide YouTube Demo Video 1
    3. Script for Live YouTube Demo Video 2
      1. Background
      2. Script
      3. Data-Driven Farming, January 25 - March 18, 2016
    4. Slides
      1. Slide 1 Farm Data Dashboards: USDA and Microsoft Innovation Challenge
      2. Slide 2 Dr. Brand L. Niemann
      3. Slide 3 USDA Data Science MOOC: AMS Geo US Farmers Markets Spotfire
      4. Slide 4 Teaching Farmers to Work with Big Data
      5. Slide 5 Week 7: Project Logs
      6. Slide 6 USDA, Microsoft to Launch “Innovation Challenge”
      7. Slide 7 Requirements
      8. Slide 8 Results
      9. Slide 9 USDA Farm Data Dashboard
      10. Slide 10 Spotfire Farm Data Dashboard
  3. Spotfire Dashboard
  4. Story
    1. Big Data Science for Precision Farming Business
    2. Slides and Videos
    3. Introduction
      1. Slide 1
      2. Slide 2
      3. Slide 3
    4. Week 1: Business Understanding
      1. Slide 4
      2. Slide 5
      3. Slide 6
      4. Slide 7
      5. Slide 8
      6. Slide 9
      7. Slide 10
    5. Week 2: Data Understanding
      1. Slide 11
      2. Slide 12
      3. Slide 13
      4. Slide 14
      5. Slide 15
      6. Slide 16
      7. Slide 17
      8. Slide 18
      9. Slide 19
      10. Slide 20
    6. Week 3: Data Preparation
      1. Slide 21
      2. Slide 22
      3. Slide 23
      4. Slide 24
      5. Slide 25
      6. Slide 26
      7. Slide 27
      8. Slide 28
      9. Slide 29
      10. Slide 30
      11. Slide 31
      12. Slide 32
      13. Slide 33
      14. Slide 34
    7. Week 4: Modeling
      1. Slide 35
      2. Slide 36
      3. Slide 37
      4. Slide 38
      5. Slide 39
      6. Slide 40
      7. Slide 41
      8. Slide 42
      9. Slide 43
      10. Slide 44
      11. Slide 45
      12. Slide 46
      13. Slide 47
    8. Week 5: Evaluation
      1. Slide 48
      2. Slide 49
      3. Slide 50
      4. Slide 51
      5. Slide 52
      6. Slide 53
      7. Slide 54
      8. Slide 55
      9. Slide 56
      10. Slide 57
      11. Slide 58
      12. Slide 59
      13. Slide 60
      14. Slide 61
      15. Slide 62
      16. Slide 63
      17. Slide 64
    9. Week 6: Deployment
      1. Slide 65
      2. Slide 66
      3. Slide 67
      4. Slide 68
      5. Slide 69
      6. Slide 70
      7. Slide 71
      8. Slide 72
      9. Slide 73
      10. Slide 74
      11. Slide 75
      12. Slide 76
      13. Slide 77
      14. Slide 78
      15. Slide 79
      16. Slide 80
      17. Slide 81
      18. Slide 82
      19. Slide 83
      20. Slide 84
      21. Slide 85
      22. Slide 86
    10. Week 7: Project
      1. Slide 87
      2. Slide 88
      3. Slide 89
      4. Slide 90
      5. Slide 91
      6. Slide 92
      7. Slide 93
    11. Week 8: Final Exam
      1. Slide 94
  5. Spotfire Dashboard
  6. USDA, Microsoft to Launch “Innovation Challenge”
  7. USDA and Microsoft Innovation Challenge for Food Resilience
    1. YouTube Video
    2. Transcript
    3. Help build a sustainable US food system by putting USDA data into the hands of farmers, researchers, and consumers
      1. Eligibility
      2. Requirements
      3. Judges
        1. Dr. Debra Peters
        2. Vince Breneman
        3. Rick Mueller
        4. Lucas Joppa
        5. Daron Green
        6. Dr. Brian Lutz
      4. Judging Criteria
      5. View All Dates
      6. Prizes
      7. How To Enter
    4. Beginning Farmer and Rancher Opportunities
    5. Technology is king down on the farm
    6. Participants
    7. Rules
    8. Submissions
    9. Updates
    10. Discussions
    11. Resources & Data
      1. Required USDA Datasets
      2. To easily access the data via Azure, visit the Farm Data Dashboard 
        1. NASS (Quickstats)
        2. ARMS
        3. CropScape
        4. VegScape
        5. Use cases
        6. AgMIP
        7. Documentation
        8. Download
      3. Support & Documentation
      4. Additional Resources
      5. Azure Resources
      6. Getting Started with an Idea
      7. Requirements for Bonus Categories
        1. Best Student-made App Award
        2. Best Visualization in Time or Space Award
        3. Best Open Source Application Award
      8. More Questions?
    12. FAQ
  8. Story
  9. Slides
    1. Slide 1 USDA Big Data Science for Precision Farming With FarmLogs
    2. Slide 2
    3. Slide 3
    4. Slide 4
    5. Slide 5
    6. Slide 6
    7. Slide 7
    8. Slide 8
    9. Slide 9
    10. Slide 10
    11. Slide 11
    12. Slide 12
    13. Slide 13
    14. Slide 14
    15. Slide 15
    16. Slide 16
  10. Research Notes
    1. Upload Shapefiles
    2. Google: fairfax county gis shapefiles for agriculture
    3. Data and Statistics
    4. Future Vision for the Use of Remotely Sensed Data
    5. Organic Farming Survey
  11. FarmLogs App Helps Farmers Collect and Retrieve Data
    1. Bushels and Bytes: The Data-Driven Farm
    2. Farmers Flying Drones May Soon Be Given Clearance
    3. Working the Land and the Data
    4. Correction: June 24, 2015 
    5. Note: June 25, 2015
  12. Google didn’t lead the self-driving vehicle revolution
  13. Drought Is the Mother of Invention on the Farm
  14. NEXT

Story

Five Data Points You Need to Know for Farming

Hello, my name is Brand Niemann, former Senior Enterprise Architect & Data Scientist with the US EPA. I work as a data scientist, produces data science products, and publishes data stories for Semantic Community, AOL Government, & Data Science & Data Visualization DC. I founded and co-organize the Federal Big Data Working Group Meetup and founded Semantic Community LLC in 2010 and am the Director and Senior Data Scientist.

I was asked by the USDA and HeatSpring to write an article based on my experience developing an online course entitled Data-Driven Farming which you can preview now free online.

My five points for you as a practicing farmer, high-school or college student in agriculture, or practicing data scientist are as follows:

  1. Farming has become more efficient and productive through precision agriculture practices.
  2. The next evolution in precision agriculture is data science.
  3. Data science for precision farming requires the integration and interpretation of many data sources using what is called a data science platform in the cloud.
  4. I have pioneered the use of a data science platform in the cloud for the integration and interpretation of many data sources for the USDA.
  5. I have now extended that work for the USDA by adding many more national and local data sources, specifically for my boyhood family farm in Nebraska City, Nebraska, to show how you can do this yourself without the help of a service provider unless you want to turn your proprietary farm data over to others.

I hope you will preview this online course and enroll.

The highlights of what you will learn in each weekly unit are as follows:

 

Week Specifics and Questions Slides, Video and Web Player

Big Data Science for Precision Farming Business

1 Five Data Points You Need to Know for Farming

What are they and where can I learn about them?

Farm Data Dashboards: USDA and Microsoft Innovation Challenge

What did I do to enter this challenge?

Slides: USDA: Open Data, Big Data and Precision Ag

Slides: Farm Data Dashboards: USDA and Microsoft Innovation Challenge

Video: USDA Innovation Challenge Announcement

Farm Data Dashboard 1: Web Player

Farm Data Dashboard: Web Player

 

Week 1 Business Understanding

1 Learn about precision agriculture from a leading university

What is a predictive yield model and where does the data come from for it?

2 Read this article and learn about these companies that provide data driven services to farmers and their farms

Do you already use these services, or want to use these service, or learn how to do it you self?

Slides

Video: Data Science in Agriculture

Farm Data Dashboard: Web Player

Week 2 Data Understanding

1 InfoAg Conference 2015: That calls itself The Premier Event in Precision Agriculture

What was presented and what can we learn?

2 The Global Open Data for Agriculture and Nutrition (GODAN) initiative

How can we improve agriculture, food and nutrition with open data?

Slides

Videos: International Plant Nutrition InstituteUSDA Innovation Challenge Announcement, and Technology Evolution in Agriculture

The Center for Open Data Enterprise: Web Player

Farm Data Dashboard: Web Player

Week 3 Data Preparation

1 UNL Crop Watch and Precision Agriculture

What does data science on UNL Crop Watch and Precision Agriculture content look like?

2 Understanding and Assessing Climate Change: Implications for Nebraska

What should I know and do about climate change as a Nebraska farmer?

 

Slides

Video: Understanding and Assessing Climate Change: Implications for Nebraska

Farm Data Dashboard: Web Player

 

Week 4 Modeling

1 Modeling the content of the InfoAg Conference 2015 for use and reuse

How many presentors had slides and how can I mine and store the content of those slides?

2 NuGIS: Nutrient Use Geographic Information System

What does their data say about statistical modeling?

Slides

Slides: InfoAg 2015 Session: Science of Precision Ag

Videos: International Plant Nutrition Institute

NUGIS: A Nutrient Use Geographic Information System For the U.S.: Web Player

 

Week 5 Evaluation

1 Evaluate USDA Big Data

What are the USDA Big Data Sets and how can I use them?

2 Evaluate the work of the biggest agro-chemical company in the world. Monsanto, and several other notable precision agriculture activities

What does Monsanto and others do and how can I use it?

Slides: Data Science for USDA Big Data

Video: Agriculture's Role in Mitigating Global Challenges

NCSS Soil Survey: Web Player

NCSS Soil Characterization Database: Web Player

 

Week 6: Deployment

1 Your Farm Data Dashboard

What are your results for: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and
Deployment?

2 Applying Your Farm Data Dashboard

Can you use this going forward?

Slides

Farm Data Dashboard: Web Player

Week 7: Project Report Link to Your Word Document Links to Your Slides, Spreadsheet, Video and Dashboard
Week 8: Final Examination

Build Your Own Questions and Answers Spreadsheet

 

Sample Question: According to a recent survey, 50% or more of farmers use data from their farm to:
Responses:
  • A  compare hybrid/variety productivity
  • B  optimize soil nutrient levels
  • C  minimize input cost
  • D  plan input purchase
Correct Answer: A

Big Data Science for Precision Farming Business

Second Call for Papers for 13th ICPA: http://www.ispag.org/ICPA/

The International Society of Precision Agriculture (ISPA) is a non-profit professional scientific organization. The mission of ISPA is to advance the science of precision agriculture globally.

The USDA and HeatSpring helped me develop an online course entitled Data-Driven Farming.

This online course provides Five Data Points You Need to Know for Farming:

  • Farming has become more efficient and productive through precision agriculture practices.
  • The next evolution in precision agriculture is data science.
  • Data science for precision farming requires the integration and interpretation of many data sources using what is called a data science platform in the cloud.
  • I have pioneered the use of a data science platform in the cloud for the integration and interpretation of many data sources for the USDA.

I have now extended that work for the USDA by adding many more national and local data sources, specifically for my boyhood family farm in Nebraska City, Nebraska, to show how you can do this yourself without the help of a service provider unless you want to turn your proprietary farm data over to others.
This presentation will provid highlights of the upcoming online course (January 25-March 18, 2016) and includes the use of the IPNI AgriStats Data Sets which were imported from multiple spreadsheets into a Spotfire Dashboard, like the other USDA, etc. data sets.

The results are (not earth shattering) at: http://semanticommunity.info/Data_Sc...ding#AgriStats

And are:

  • Population has increased while lands (Agricultural, Arable, Forested, and Irrigated) have remained the same or even decreased!
  • Nutrient Use for our main crops (Cotton Lint and Seed, Maize, Potatoes, Rice, paddy, and Wheat) exceeds Nutrient Removal from the Best Fit Line, but not for some individual crops and years.

But collecting and using data to improve profitability is the future of farming, but it can be overwhelming to farmers, students, and even data science professionals. This overview of the hands-on course, developed in partnership with the USDA, will help farmers implement a data-driven approach to decision making.

The presentation also includes a live demo of the author's USDA-Microsoft Innovation Challenge entry: http://devpost.com/software/farm-data-dashboards 

Keywords: Data-Driven Farming, USDA, Data Science, IPNI AgriStats, Spotfire Dashboards

Thank you for your ICPA abstract submission, #1753.

This email is to notify you that you were included as the corresponding author for an abstract submitted Nov 12, 2015 to the 13th ICPA.

The abstract was submitted by Niemann and titled, "Big Data Science for Precision Farming Business".

All future correspondence regarding this abstract will be sent to Brand Niemann at bniemann@cox.net. If you have questions about this submission, please contact Niemann or abstracts@ispag.org.

The ICPA Scientific Committee will review the abstract for suitability based on scientific content and clarity. Abstracts meeting these criteria will be accepted for oral or poster presentation at the Conference.

Authors of accepted abstracts will be entitled to present their research at the Conference after payment of registration fees. They will also be entitled to submit full papers for the Conference Proceedings in February of 2016 (more details to follow).

You will be notified regarding the status of your abstract as soon as the review process is completed. If you have any questions, please contact us at abstracts@ispag.org with your unique Abstract ID #1753.

Sincerely,
Dr. Ken Sudduth
Chair of the 13th International Conference on Precision Agriculture 
President, International Society of Precision Agriculture

Smarter Farming with USDA Data

Source: https://www.govloop.com/smarter-farm...ith-usda-data/

I grew up on a farm in Chunchula, Alabama. It was a small operation – a few goats, turkeys, and chickens; as well as a small orchard and a few crops. The most data we collected on productivity was how many eggs I could carry back to the house in my basket on a given day. If you asked my Papa to hook up a computer or tablet to his tractor, he probably would have told you to get away from his tractor.

Fast forward twenty years and things are very different. My family lives in a city and the equipment we used to farm is wildly outdated. That’s no surprise, according to Joyce Hunter, Acting Chief Information Officer within the Deputy Chief Information Office of Policy and Planning at USDA. 

Changing the Way America Farms

Today, farming equipment is as sophisticated as your smart phone. “Many people think farmers, green acres, and the Beverly Hillbillies,” joked Hunter. “But you have to go to a farm these days. It’s absolutely amazing!” Drones are used to monitor farmlands, tractors are connected to the internet via sensors, and computers execute many of the daily tasks required to run a farm.

However, there are fewer farmers using this equipment. According to Hunter, only 2% of the US population is involved in farming. At the same time, the population is growing exponentially. By 2050, USDA estimates that the United States will need to feed 9-10 billion people worldwide.

In order to meet that demand with a reduced workforce and less farmland, USDA is harnessing the power of data analytics.

Enabling Precision Agriculture

Ultimately, USDA wants to harness the power of all these data points created by connected farming equipment, drones, and even satellites to enable precision agriculture. Precision farming, defined simply, is planting the rights crops in the right places at the right time in order to get the fastest, highest yield without producing substantial food waste.

Hunter calls precision agriculture the “holy grail” because it’s the means by which we will solve the food supply and demand imbalance that we currently face. Fewer farmers will be able to produce more of the right food, in shorter amounts of time.

To achieve that precision, farmers need a lot of data to inform their planting strategies. That’s why USDA is investing in big, open data projects. “We hope open data and big data will combine together to provide farmers and consumers just the right kind of information to make the best decisions,” said Hunter.

Digging into USDA Data

How will USDA do it? Hunter reminded us that her agency is the oldest federal civilian agency, dating back to the Civil War, and they have been collecting data since that time. But now, they are doubling down on organizing and processing that data. The department is also leveraging new data sources, including the scientific instruments, sensor technologies, and mobile devices used on farms.

Most importantly, USDA is diligently working to get that data into the hands of farmers. The agency is the cornerstone of the Global Open Data for Agriculture and Nutrition (GODAN) Initiative, which supports efforts to make agriculturally and nutritionally relevant data available for public global use. So far, the project has published over 500 data sets via the federal website data.gov.

Additionally, USDA is supporting multiple civic hackathons by providing agricultural data and the agency is creating an agricultural domain within the Department of Homeland Security’s National Information Exchange Model (NIEM) to support a common set of data standards across the industry.

Finally, the department contributes to multiple specific open data projects. Hunter offered three examples:

  • Office of the Secretary Disaster Clearinghouse Project – Developing a one-stop application for disaster victims to find assistance before, during and after a disaster
  • Census Find it and Connect it Project – Running a pilot using USDA, NOAA and Census data to determine the economic impact of a flood on Consumers in Johnstown, PA.
  • Census One City, One Problem Software Development Kit (SDK) Project – Enabling cities and counties to use data to deliver tangible and transformative results for their communities

All of these efforts combine to support the USDA Open Data Initiative, called OpenAg. It’s goal, according to Hunter, is to “expand and promote a culture of Open Government among the Department by making its information resources widely accessible, discoverable, and usable by Federal government agencies and by the general public.”

From what we learned today, USDA is well on its way to achieving that mission. Furthermore, the agency is taking innovative steps to move their data initiatives even further. Earlier this week, USDA and Microsoft announced an Innovation Challenge to award a total of $60,000 for creative ideas to leverage data to build smarter farming systems.

For more information on USDA’s data initiatives, be sure to check out Joyce Hunter’s full presentation by clicking here.

Photo Credit: Flickr/U.S. Department of Agriculture

Thank you for attending GovLoop's event: Data Done Smart: Your Guide to Analytics in Government yesterday. We hope you left with a great deal of knowledge and resources.

In order to obtain your CPE, you must complete the post-event evaluation before 2:00 PM ET Monday, August 3rd. Once completed, you’ll receive your CPE certificate by the end of next week.

For a refresher on how data analytics are transforming government provided by our jam-packed panels, be sure to check out the blog posts from the event below and share with a colleague:

Data Analytics 2015 Recap Posts

Want more information? Download our guide: The Open Data Playbook For Government

Slides

Slides

Slide 1 USDA: Open Data, Big Data and Precision Ag

OMB Memo 13-13 released last year, requires federal agencies to develop and release data that can be used by other government agencies,  industry, academia, and the public.

It also requires Agencies to develop and maintain and enterprise data inventory that can be Expanded, Enriched and is Open.

It requires Agencies to make their data machine readable and using open formats and data standards for new information development and collection.   It requires Agencies to use open licenses and establishes processes that will identify information that can not be release for various reasons.  (privacy, legal, security)

JoyceHunter07302015Slide1.PNG

Slide 2 USDA Open Data 1

JoyceHunter07302015Slide2.PNG

Slide 3 USDA Open Data 2

JoyceHunter07302015Slide3.PNG

Slide 4 Open Data Projects

JoyceHunter07302015Slide4.PNG

Slide 5 Intersection of Open Data, Big Data and Open Government

JoyceHunter07302015Slide5.PNG

Slide 6 What is Precision Ag?

JoyceHunter07302015Slide6.PNG

Slide 7 Precision Ag and Big Data

JoyceHunter07302015Slide7.PNG

Slide 8 What Is Generating Big Data?

JoyceHunter07302015Slide8.PNG

Slide 9 Value of Precision Ag Data

JoyceHunter07302015Slide9.PNG

Slide 10 Decision Support for Farmers

JoyceHunter07302015Slide10.PNG

Slide 11 Farmer Profitability Strategy

JoyceHunter07302015Slide11.PNG

Slide 12 Precision Ag Delivers…

JoyceHunter07302015Slide12.PNG

Slide 13 Smarter Farming with USDA Data

https://www.govloop.com/smarter-farm...ith-usda-data/

JoyceHunter07302015Slide13.PNG

Slide 14 Questions

JoyceHunter07302015Slide14.PNG

Story

Farm Data Dashboards: USDA and Microsoft Innovation Challenge

1 Project name: Farm Data Dashboards 

Links to Live Application on the Web: Farm Data (Dashboard 1 Web Player (ARMSDB Survey Data 1: USDA Crop Production Processes, ARMSDB Survey Data 2: Farm Financial Practices) and Farm Data Dashboard 2 Web Player (Commodity Value-NASS QuickStats)

See: Video 1 Slide Demo (Camtasia): YouTube Video 2 Live Demo (Camtasia): YouTube 

Also see below: Script for Slide YouTube Demo Video 1 and Script for Live YouTube Demo Video 2

2 Tagline: USDA Data Science MOOC, Teaching Farmers to Work with Big Data, & Data Science Audit Results

3 Thumbnail image

USDAMOOC-Spotfire-AMS-Geo US Farmers Markets.png

Teaching Farmers How To Work With Big Data - Week 7 Project Logs.png

FarmDataDashboard1-Spotfire.png

4 Which tools and resources did you use?

5 Links (Project Web Site, File Repository, and GitHub/DataHub):

6 Video 1 Slide Demo (Camtasia): YouTube Video 2 Live Demo (Camtasia): YouTube

7 Image gallery

BrandNiemann07272015Slide1.PNG

8 Teaching Farmers to Work with Big Data

  • HeatSpring, an online course provider, found my USDA Data Science MOOC, and asked me to do an 8-week Data-Driven Farming Course (October 26-December 18, 2015).
  • This online course uses the Tools and Data Sets in US Climate Resilience Toolkit (Taking Action Case Studies of Resilience in Action) to help farmers and data scientists plan for how current and future climate changes might influence food systems.
  • The online course uses the Cross Industry Standard Process for Data Mining (CRISP-DM) and shows farmers how to develop a Farm Data Viewer and Farm Data Decision Dashboard.

9 USDA, Microsoft to Launch “Innovation Challenge”

Top Prize of $25K Offered for Most Creative Application on Food Resiliency and Data Science.

WASHINGTON, July 27, 2015 - The U.S. Department of Agriculture is partnering with Microsoft to launch the “Innovation Challenge,” a competition to develop software applications that help farmers, agriculture businesses, and consumers explore how climate change will affect their food systems.

Entrants are invited to develop and publish new applications and tools that can help users analyze multiple sources of information, including key USDA data sets.

The Food Resilience theme of the challenge seeks to inspire the creation of tools that help users analyze and explore our food systems.

For more than 100 years, USDA has compiled data from economic reports, satellite imagery, remote sensors, and surveys on crop production practices, financial outcomes, demographics and remote sensing products on the health of crops around the country. 

10 Requirements

  • What to Create: Submit a working, interactive application that integrates one or more of the required USDA datasets. 
    • Static data visualizations will not be eligible. Applications must include interactive functionality (e.g. the user can change parameters to update the visualization and/or result). My Note: I use interactive data visualization functionality
  • Eligible Platforms:
    • Smartphone or tablet (iOS, Android, Blackberry, Kindle, Windows 8 Mobile) My Note: Yes
    • Web (mobile or desktop) My Note: Yes
    • Desktop (Windows PC, Mac Desktop) My Note: Yes
    • Software running on other publicly available hardware (including, but not exclusive to, wearable technology, open source hardware, etc.) My Note: Yes
  • Supplemental Material: You must submit a demo video (hosted on YouTube, Vimeo, or Youku) that walks through the main functionality of the application via screencast or video. You must also submit a text description and at least one image/screenshot of your working application. My Note: Yes
  • Testing: You must make your app available for testing by providing a link to access your installation file, an uploaded installation file, a beta distribution build, etc. See full testing access options. My Note: Yes
  • New & Existing Solutions: Apps may be newly created or pre-existing. If the submitted app existed prior to the competition’s submission start date, it must have been updated to integrate the required USDA data during the submission period. My Note: Yes

11 Results

  • Data Science Audit of the USDA and Microsoft Innovation Challenge for Food Resilience:
    • Comments transmitted to USDA CIO and CDO and Captured in Online Knowledge Base. See Audit Below
  • USDA Data Science MOOC, NASS QuickStats 2.0, and Big Data Science for Precision Farming:
    • Data Science Data Publications with Knowledge Bases, Excel Spreadsheets, and Spotfire Analytics and Visualizations. 
  • USDA Farm Data Dashboard-Spotfire Farm Data Dashboard Comparisons:
    • USDA Eight Tabs Versus Spotfire 25 Tabs.
    • ARMS 7 Filters Versus Spotfire 9 Filters.
    • ARMS Get or Download Data Versus Spotfire See All 35,928,742 Rows by 14 Columns of the Data!
    • ARMS Has Number of Records Versus Spotfire Has Temporal and Spatial Number of Records and Visualizations!

Data Science Audit Results

I reproduced your Farm Data Dashboard: ARMS in Spotfire (see screen capture in Slide 10 above) and found some big differences (e.g. Records Available USDA FD Dashboard (12,140,959) and records imported in Spotfire (37,928,742) from downloaded  database.

USDA-Spotfire Farm Dashboard Comparisons.png

I think competition developers will have a difficult time finding an open source tool that will do what Spotfire does with that many records.

I also could not match three of the FD Dashboard categories to the categories in the downloaded database and it's PDF documentation file.

This will also make things difficult for the competition developers.

I complete my Data Science Audit of the challenge content and my results are at:

Some highlights are:

To be eligible for the Best Open Source Application Award, you must make your submitted application code available to the public at the time of entry under one of the approved open source licenses. I am not using Open Source and I am not Coding because I have found a commercial application (TIBCO Spotfire) that makes it possible to do state-of-the-art data science analytics and visualizations without coding, although it supports coding if I want to use it.

I only use the entire data set and not the APIs.

Not sure how and why a farmer or data scientist uses hugh raster data from CropScape and VegScape.

The Farm Data Dashboard for CropScape and VegScape open to blank screens at least on my computer.

The Farm Data Dashboard Uses Cases: I looked at these 5 PDF files do not see how or why I would use them.

The Cropland Data Layer Metadata link contains a link at the very bottom of the page to Visit a Bioenergy/CDL web mashup Bioenergy Knowledge Discovery Framework/U.S. Dept. of Energy. Is this the tie in to the Best Map Overlay suggestion to show the number of pixels of corn planted on different soil types in Nebraska, or the distribution of distance between cornfields and bioenergy plants in the Midwest? If so, it should be much easier to find.

The Cropland Data Layer Metadata link FAQ explains:

The geospatial data product called the Cropland Data Layer (CDL) is hosted on CropScape. The CDL is a raster, geo-referenced, crop-specific land cover data layer created annually for the continental United States using moderate resolution satellite imagery and extensive agricultural ground truth.

NASS developed both the CropScape and VegScape web services in cooperation with the Center for Spatial Information Science and Systems, George Mason University, Fairfax, VA.

There will be differences between CropScape and official NASS estimates when comparing acreage statistics at the state, district, and county levels. Statistics generated by CropScape are dependent upon pixel counting. Pixel counting is usually downward biased when compared to the official estimates. Counting pixels and multiplying by the area of each pixel will result in biased area estimates and should be considered raw numbers needing bias correction.

Now I better understand what the folks at GMU have been doing for the USDA.

Bottom Line: I can add the 14 spreadsheets in the ERS ARMS Farm Financial and Crop Production Practices to my Online Course and Farm Data Dashboard on Data-Driven Farming, and see if I can figure out if there is a way to include CropScape and VegScape data.

Unfortunately, I think I fail this criteria:

(iv) New & Existing: Applications must be either newly created by the Maker or, if the Application existed prior to the Challenge Submission Period, must have integrated the required USDA data after the start of the Challenge Submission Period.

because I did my work with your USDA data sets before July 24th.

I just looked through all of the Challengepost.com web pages and had these thoughts:

1.Winners Announced: On or around December 2, 2015 (2:00pm Eastern Time)-This is close to the time that my online class students would be finishing their 8 week course and delivering a project that would/should meet these requirements.

2.Nothing to see here yet. The organizers have not made submissions public.- I could submit two applications right now (the USDA Data Science MOOC and the Big Data Science for Precision Farming Business)

3.To build your app you'll need to utilize one or more of the provided datasets listed on the Challenge Resources page. You may use any tool you like to analyze or utilize this data. - So I do both the application and the USDA Data set in the Cloud so they are together, not separate like the USDA Data in the Azure Cloud and the application running in another cloud, etc. Both of my applications and their USDA data sets run in the Amazon Cloud.

In the rules (see excerpts below), I found a section of amazingly detailed requirements that I have just done, namely non-static visualizations (Spotfire does these the best), temporal and spatial with USDA and non-USDA data, "corn planted on different soil types in Nebraska", etc.

I see that Center for Rural Affairs is based in Lyons, Nebraska, close to my family farm in Nebraska City that I have used in the online class!

C. Best Visualization in Time or Space Award Category Requirements

To be eligible for the Best Visualization in Time or Space Award, a Submission must meet all of the Submission Requirements described in Section 4, and the requirements of one of the following sub-categories:

(i) Best Data Visualization of Required Data Over Time

The Application includes a visualization tool that allows users to view either: (a) trends through time for multiple variables for the same location, or (b) the same variable(s) across a spatial extent (the entire US, or a region). Makers are encouraged to combine the USDA data with other data.

(ii) Best Map Overlay

The NASS cropscape, vegscape, and agriculture census maps (and underlying data), and ERS atlases (maps with data) are all mapped at the county level. The Application includes map overlays at the county or state level, and conducts analyses of these map overlays. For example, the analysis could show the number of pixels of corn planted on different soil types in Nebraska, or the distribution of distance between cornfields and bioenergy plants in the Midwest.

(iii) Best App Including Crowdsourced Data

The Application includes a real-time component by integrating the ability to capture and integrate crowdsourced data in addition to USDA data.

I am running my Big Data Science for Precision Farming Business application in the Amazon Cloud.

Script for Slide YouTube Demo Video 1

YouTube

Slide 1 Hello, this is Dr. Brand Niemann, Director and Senior Data Scientist with Semantic Community and Founder and Organizer of the Federal Big Data Working Group Meetup, here to tell you about my Data Science for Farm Data Dashboards for the USDA and Microsoft Innovation Challenge.

The first 3 links at the bottom of this slide are to background information and the last 2 links are to this project.

Slide 2 This slide tells you that Brand Niemann is a data scientist / data journalist who grew up working on a farm in Nebraska. He was asked by the USDA to create awareness and access to USDA data in partnership with their Open Data Communications Plan. He created the USDA Data Science MOOC (Massive Open Online Course). Brand is a former Senior Enterprise Architect and Data Scientist with the US EPA, who organizes the Federal Big Data Working Group Meetup - a meetup that regularly attracts hundreds of the top data scientists in and around Washington D.C. He has worked as a data scientist for a number of organizations, produced data science products for a large number of data sets, and published data stories for Federal Computer Week, Semantic Community and AOL Government.

Slide 3 This slide is a screen capture of the USDA Data Science MOOC Spotfire Visualization for the USDA AMS Geo US Farmers Markets. These visualizations (Bar Chart, Map, Data Table, and Filters) are dynamically linked and fully interactive. One interesting example is to filter out the Farmers market the do not accept SNAP and see the green dot location on the map disappear. About 3 times as many Farmers Markets do not accept SNAP as those that do.

Slide 4 My Teaching Farmers to work with Big Data - Data Driven Farming Online Course, to be taught in the fall, uses the Tools and Data Sets in US Climate Resilience Toolkit (Taking Action Case Studies of Resilience in Action) to help farmers and data scientists plan for how current and future climate changes might influence food systems.

The online course also uses the Cross Industry Standard Process for Data Mining (CRISP-DM) and shows farmers how to develop a Farm Data Viewer and Farm Data Decision Dashboard.

Slide 5 This slide is a screen capture of the Farm Data Dashboard Spotfire Visualization for Week 7: Project Logs. This enables the farmer to log and display daily climate and farm business activities.

Slide 6 This slide summarizes the highlights of the July 27th USDA Press Release: USDA, Microsoft to Launch “Innovation Challenge”. My application uses both historic and current climate and farm business data sources to document how climate changes have and will affect their food systems.

Slide 7 This slide summaries how I have met the challenge requirements.

Slide 8 This slide summarizes my results. I have done a Data Science Audit of the USDA and Microsoft Innovation Challenge for Food Resilience and transmitted my comments transmitted to the USDA CIO and CDO and have captured them in my Online Knowledge Base.

I have created the USDA Data Science MOOC, NASS QuickStats 2.0, and Big Data Science for Precision Farming that consist of
Data Science Data Publications with Knowledge Bases (2), Excel Spreadsheets (3 with multiple tabs), and Spotfire Analytics and Visualizations (3 with multiple tabs). 

I have made comparisons between the USDA Farm Data Dashboard and the Spotfire Farm Data Dashboard, which are summarized in my Data Science Audit and published to the Challenge Post and Online Knowledge Base, with questions and concerns about the USDA Farm Data Dashboard Data Sets and Documentation.

Slide 9 This slide is a screen capture of the USDA - Microsoft Challenge Post Farm Data Dashboard which delivers multiple data sets with filtering and data downloads.

Slide 10 This slides is a screen capture of the Spotfire Farm Data Dashboard which delivers not only multiple data sets with filtering and data downloads, but also temporal and spatial analytics and visualizations, as requested in the challenge. The Web Player version is hosted in the Amazon Cloud and is fully interactive with dynamically linked adjacent visualizations of both the data and the graphics. It is fully interactive with the temporal and spatial visualizations changing as one or more filters on the right side are changed.

Thank you for your attention. This concludes the presentation and demonstration.

Script for Live YouTube Demo Video 2

Demo

Background

On a related note, last night we had a project night for the USDA Challenge, and one question that arose is whether there is a "data dictionary" for any of the datasets.   That is, beyond the variable names, what do the values for certain variables mean (in cases when its not intuitive).  Do you know if such a "data dictionary" exists?  I can ask them members of the group more on this if the above isn't a specific enough question.

Excellent question and one that I have asked Serena of DevPost and the Microsoft and USDA  people involved in this Challenge about.

Please see my Data Science Audit of the USDA Challenge.

http://semanticommunity.info/Data_Science/Big_Data_Science_for_Precision_Farming_Business#Data_Science_Audit_Results

A related, follow-up question:  are there database schemas so as to understand the primary keys to help with joins?  or are these datasets flat files for the most part?

No worries if not as we can construct schemas using the various online resources; though just thought to ask as this also came up in our project night.  I do not know recall specifically what tables/fields were being discussed when this was asked, though can ask the group if this would help.  Thinking about this more, parameters which fall into the time or space categories perhaps are fairly easy to join, and those may end up being the ones that we pivot around the most.

Flat files. I integrated their two largest data sets and showed the temporal and spatial variations together using Spotfire. I have not seen anyone else do that.

I should be more specific:

Script

There are two questions to be answered: (1) Have you explained how your app or data visualization helps support farmers and build a sustainable U.S. food supply? (2) Have you demonstrated your integration of one of the required USDA data sets?

First question (2): There are three large data sets (ARMSDB Survey Data 1: USDA Crop Production Processes, ARMSDB Survey Data 2: Farm Financial Practices, and Commodity Value. The first two I integrated in my FarmDataDashboard-Spotfire1, for which the columns are identical as you can see below.

Farm Data Dashboard Script 1.png

 

These two data sets are displayed both temporally and spatially in the same Spotfire visualization below where they can be filtered by one or more of the 13 columns.

Farm Data Dashboard Script 2.png

 

The Spotfire visualization below shows my comparison of the original USDA farm Data Dashboard and  what I when I downloaded and imported those data sets in Spotfire and tried to create a data dictionary.

Farm Data Dashboard Script 3.png

 

Finally, I wanted to integrate the third data set (Commodity Value) and show how it helps support farmers and build a sustainable U.S. food supply in my FarmDataDashboard-Spotfire2. This is also called the NASS Quick Stats data set which is massive in size (1.6 TB) with over 31,000,000 rows and 35 columns. Displaying all of this data in memory on the Web is a major accomplishment in itself and show the power of Spotfire. As you can see below, these data have columns counts that vary from 0 to over 31,000,000, time from 1872 to the present, and states, regions, and ZIP codes as shown below.

Farm Data Dashboard Script 4.png

Farm Data Dashboard Script 5.png

So every statistics that the USDA NASA has every collected is available by filtering by the 34 columns of the over 31 million rows!

But there is no cross-walk between Commodity Vale and the first two data sets (ARMSDB Survey Data 1 and ARMSDB Survey Data 2).

All of these details are at: http://semanticommunity.info/Data_Science/Big_Data_Science_for_Precision_Farming_Business

And all these details will be part of my online course early next year: https://www.heatspring.com/courses/data-driven-farming See Below.

I am also going to create a 5 minute video of what I have just explained above that I will post and send a link to you (John P. Verostek, National Data Science Organizers Workshop, November 5-6, 2015).

Thanks. I appreciate the additional info on the USDA datasets and Spotfire.  It is a lot of data to pour through, and I can see how Spotfre can help out in this respect.

Data-Driven Farming, January 25 - March 18, 2016

https://www.heatspring.com/courses/data-driven-farming

Instructor: Brand Niemann

Price: $1295

This is a hands-on course for farmers, developed in partnership with the USDA. Spend eight weeks collecting and working with the most useful data for your farm.

View Course

Did you know?

Data-Driven Farming Quick Fact: Self-Paced Online Course

This course is self-paced and entirely online. The lecture videos and other materials are available any time, and the instructor responds to student questions through a discussion board. In order to participate, you will need an Internet connection and an up-to-date web browser.

Slides

Slides

Slide 1 Farm Data Dashboards: USDA and Microsoft Innovation Challenge

Semantic Community

Data Science

USDA Data Science MOOC

Big Data Science for Precision Farming Business

Link to Live Application on the Web: Web Player

BrandNiemann07272015Slide1.PNG

Slide 2 Dr. Brand L. Niemann

BrandNiemann07272015Slide2.PNG

Slide 3 USDA Data Science MOOC: AMS Geo US Farmers Markets Spotfire

Web Player

BrandNiemann07272015Slide3.PNG

Slide 4 Teaching Farmers to Work with Big Data

BrandNiemann07272015Slide4.PNG

Slide 5 Week 7: Project Logs

Slide 93

BrandNiemann07272015Slide5.PNG

Slide 6 USDA, Microsoft to Launch “Innovation Challenge”

http://usdaapps.challengepost.com

BrandNiemann07272015Slide6.PNG

Slide 7 Requirements

BrandNiemann07272015Slide7.PNG

Slide 9 USDA Farm Data Dashboard

Farm Data Dashboard

BrandNiemann07272015Slide9.PNG

Slide 10 Spotfire Farm Data Dashboard

Web Player

BrandNiemann07272015Slide10.PNG

Spotfire Dashboard

For Internet Explorer Users and Those Wanting Full Screen Display Use: Web Player Get Spotfire for iPad App

Error: Embedded data could not be displayed. Use Google Chrome

Story

Slides

Big Data Science for Precision Farming Business

HeatSpring, an online course provider, found my USDA Data Science MOOC, and asked me to do an 8-week Data-Driven Farming Online Course (October 26-December 18, 2015). HeatSpring provides an online tool and Basecamp for the instructor and students.

I created the course description, bio, and tool (Excel spreadsheet of Experts from Week 6 Project Assignment).

I am testing the course on my Federal Big Data Working Group Meetup on September 14th. A Meetup member, Peter Morosoff, got me interested in this subject by sending me three NY Times articles:

The most important decision is the goals and focus of the online course to level-set expectations for the students. I decided on the following: You can use this course to accomplish any (or all) of these goals:

  • Your Goals with Data and Entrepreneurship
  • Sources of Farm Data and Precision Farming Success
  • Unlock Your Data and Choose Your Path
  • Finding and Using Soils Data
  • Farm Data Viewer
  • Farm Data Decision Dashboard
  • Three Choices for Your Project/Deployment

I want to make the focus clear: I help you find, capture, store, and display farm data so you can decide how to use it yourself, and with expert help, if you so desire, to increase farming success. The attached template is a compilation of experts. That alone is worth the price of the course!

I decided to use the Cross Industry Standard Process for Data Mining (CRISP-DM) as the course framework so students would go from their farm business understanding, to data understanding, to data preparation, modeling and evaluation, to actually deploying an application they could continue to use in their daily work to improve farm productivity and profitability. That deployed application would be their class project which 50% of the credit (25% for class exercises and 25% for final exam to determine pass or fail at 75% or more).

I also wanted the class to drill down from national and regional farm data to local and individual farm business data, a daunting task for an only 8-week online course, so I picked Nebraska City, Nebraska, where the Niemann Family Farm was that I grew up on in the summers as young boy.

As a data scientist/data journalist with 30+ years of Federal Government experience with the US Environmental Protection Agency, where I was a data architect/senior enterprise architect/data scientist, I produced environmental data dashboards to I could see, manipulate, and visualize all the data right before me, without having to go looking around web sites, files, etc. every time.

So I want to provide farmers with an example of how to do that, by providing a set of narrated slides that show them farm data sources, an Excel spreadsheet where multiple farm data sources are captured, and a Spotfire dashboard where those farm data sources are analyzed and visualized. The student can use their data storage and visualization tool(s) of choice, because this online course is not about technology, but about farm data.

This course could not possibly show all the farm data for all the different types of farming and all the different places, etc. This online course is not about "boiling the ocean" of Data-Driven Farming, but more about Big Data Science for Precision Farming Business. Data Science is the study and use of data to better inform decisions. Big Data Science is when you have multiple data sources from multiple places in multiple formats, that need to be integrated and analyzed. Precision Farming is when you want to do farming with additional tools and techniques like the USDA found rice farmers were using:

  • Yield Monitor
  • Yield Map
  • GPS Map of Soil Properties
  • Guidance System
  • Variable Rate Applicator for Fertilizer Applications

In my exploration for "precision farming data", I found a USDA report: Rice farms are adopting precision agriculture technologies

This chart is found in the joint ERS/National Agricultural Statistics Service (NASS) report, 2013 ARMS—Rice Industry Highlights, based on ARMS Farm Financial and Crop Production Practices data.
 
Ricefarmsareadoptingprecisionagriculture2006and2013.png

Since there are many federal and local government agencies with experts and data to help you, I gave you an Excel file that puts all of the top sources of U.S. Experts in one place, including links to their web sites. Regional and locally-focused ​​​centers ​across the nation with experts are available to help you build resilience to climate-related changes and impacts​ in your community​.​ I have digitized the information in the maps for State Climatologists, NOAA, USDA, and Department of the Interior. You can quickly search by your state and find links to the experts, their critical data sets you may need, and their wisdom in how to use those data sets.

So if you are too busy to take the online course and/or cannot afford it, you can use this spreadsheet as both a resource to get free help at your convenience and as an example of how to scrape information off the Web into your spreadsheet for handy future reference.

The most important decision you have to make about your 'data-driven,more precision' farm business (and this course tries to help with that decision) is whether or not you are going to do Data-Driven Farming on your own or with the assistance, or complete outsourcing, to one of the new startups listed below that I got from a recent article (May 19, 2015) in Forbes magazine: Farm data continues to be hot in Silicon Valley. You can see my efforts to understand in detail on of those startups: FarmLogs below. That experience did not work out well so I am next trying the Farmers Business Network.

 

Name URL Comments
Farmers Business Network https://www.farmersbusinessnetwork.com/ Unable to participate due to roadshow in Midwest
FarmLink http://www.farmlink.com/  
Adapt-N http://www.adapt-n.com/  
Farmers Edge http://www.farmersedge.ca/  
FarmLogs https://farmlogs.com/ See Story below. No service and no response
aWhere http://www.awhere.com/  
Granular http://www.granular.ag/  
Farmeron https://www.farmeron.com/  
OnFarm http://www.onfarm.com/  
Agralogics http://agralogics.com/ Email bounced back
Blue River Technology http://www.bluerivert.com/  
Precision Hawk http://www.precisionhawk.com/  
Bright Farms http://brightfarms.com/s/  
Indoor Harvest http://indoorharvest.com/  
PodPonics http://podponics.com/  
Freight Farms http://www.freightfarms.com/  

Ideally, as the instructor, I should be familiar with all of these and maybe I will have time to do that.

A very important consideration for attracting students to the online course is a marketing/teaching partner and I have three possibilities so far:

  • USDA - CIO/CDO
  • Agriculture.com - SoyRoy.com
  • Farmers Business Network

There are pros and cons for each and we are currently in discussions.

I am still building the online course at this stage so I need flexibility in adding/rearranging, etc. content in both the PowerPoint Slides, Excel Spreadsheet, and Spotfire Dashboards. I have that flexibility in this MindTouch environment I am using here. I plan on using Camtasia to record the narrated slides for each of the 8 cource weeks.

So far it appears that for my class project (Nebraska City, Nebraska), the free data sources are as good or better than what I would have to pay for from a startup.

I am also interested in what the largest agrochemical company in the world, Monsanto, offers and found:

Putting data in the hands of farmers.

Every year, farmers make decisions about growing crops. Decisions that are based on the information they have on-hand. We help them harness the power of information to make better-informed decisions and to use their resources more sustainably.

We offer a variety of tools to help farmers have better harvests, ranging from easy-to-use apps to the latest tractor technologies.

  • True to its name, The Climate Corporation provides apps to allow farmers to check weather and field conditions remotely from their phone or tablet. The convenient app can also send text alerts at the ideal time for a farmer to plant, protect or harvest their crop.
  • Our FieldScripts service translates information about a farmer’s past corn crops to help him understand how soil and water conditions vary across their land. The service then provides a customized recommendation for planting seed that will lead to a bountiful harvest.
  • When planting seeds, spacing and depth play a key role in how well a plant grows – or if it sprouts at all. Precision Planting technologies have helped ensure that seeds are planted in the best place in the soil to grow.

Armed with the information they need to succeed, today’s farmers are making smarter decisions which leads to more a sustainable farming system and having better harvests.

The DRAFT course for review consists of:

Slides

Spreadsheet

Spotfire Dashboard

Slides and Videos

Week 1: Business Understanding In-Line

Week 2: Data Understanding In-Line

Week 3: Data Preparation In-Line

Week 4: Modeling In-Line

Week 5: Evaluation

Week 6: Deployment

Week 7: Project

Week 8: Final Exam

MORE TO FOLLOW

About the Author

Brand Niemann is a data scientist / data journalist who grew up working on a farm in Nebraska. He was asked by the USDA to create awareness and access to USDA data in partnership with their Open Data Communications Plan. He created the USDA Data Science MOOC (Massive Open Online Course). Brand is a former Senior Enterprise Architect and Data Scientist with the US EPA, who organizes the Federal Big Data Working Group Meetup - a meetup that regularly attracts hundreds of the top data scientists in and around Washington D.C. He has worked as a data scientist for a number of organizations, produced data science products for a large number of data sets, and published data stories for Federal Computer Week, Semantic Community and AOL Government.

Introduction

Slide 1

BrandNiemann07182015Slide1.PNG

Slide 2

BrandNiemann07182015Slide2.PNG

Slide 3

BrandNiemann07182015Slide3.PNG

Week 1: Business Understanding

Your Goals with Data and Entrepreneurship

Slide 4

BrandNiemann07182015Slide4.PNG

Slide 5

BrandNiemann07182015Slide5.PNG

Slide 6

BrandNiemann07182015Slide6.PNG

Slide 7

BrandNiemann07182015Slide7.PNG

Slide 8

BrandNiemann07182015Slide8.PNG

Slide 9

BrandNiemann07182015Slide9.PNG

Slide 10

BrandNiemann07182015Slide10.PNG

Week 2: Data Understanding

Sources of Farm Data and Precision Farming Success

Slide 11

BrandNiemann07182015Slide11.PNG

Slide 12

BrandNiemann07182015Slide12.PNG

Slide 13

BrandNiemann07182015Slide13.PNG

Slide 14

BrandNiemann07182015Slide14.PNG

Slide 15

BrandNiemann07182015Slide15.PNG

Slide 16

BrandNiemann07182015Slide16.PNG

Slide 17

BrandNiemann07182015Slide17.PNG

Slide 18

BrandNiemann07182015Slide18.PNG

Slide 19

BrandNiemann07182015Slide19.PNG

Slide 20

BrandNiemann07182015Slide20.PNG

Week 3: Data Preparation

Unlock Your Data and Choose Your Path

Slide 21

BrandNiemann07182015Slide21.PNG

Slide 22

BrandNiemann07182015Slide22.PNG

Slide 23

BrandNiemann07182015Slide23.PNG

Slide 24

BrandNiemann07182015Slide24.PNG

Slide 25

BrandNiemann07182015Slide25.PNG

Slide 26

BrandNiemann07182015Slide26.PNG

Slide 27

BrandNiemann07182015Slide27.PNG

Slide 28

BrandNiemann07182015Slide28.PNG

Slide 29

BrandNiemann07182015Slide29.PNG

Slide 30

BrandNiemann07182015Slide30.PNG

Slide 31

BrandNiemann07182015Slide31.PNG

Slide 32

BrandNiemann07182015Slide32.PNG

Slide 33

BrandNiemann07182015Slide33.PNG

Slide 34

BrandNiemann07182015Slide34.PNG

Week 4: Modeling

Finding and Using Soils Data

Slide 35

BrandNiemann07182015Slide35.PNG

Slide 36

BrandNiemann07182015Slide36.PNG

Slide 37

BrandNiemann07182015Slide37.PNG

Slide 38

BrandNiemann07182015Slide38.PNG

Slide 39

BrandNiemann07182015Slide39.PNG

Slide 40

BrandNiemann07182015Slide40.PNG

Slide 41

BrandNiemann07182015Slide41.PNG

Slide 42

BrandNiemann07182015Slide42.PNG

Slide 43

BrandNiemann07182015Slide43.PNG

Slide 44

BrandNiemann07182015Slide44.PNG

Slide 45

BrandNiemann07182015Slide45.PNG

Slide 46

BrandNiemann07182015Slide46.PNG

Slide 47

BrandNiemann07182015Slide47.PNG

Week 5: Evaluation

Data Viewer

Slide 48

BrandNiemann07182015Slide48.PNG

Slide 49

BrandNiemann07182015Slide49.PNG

Slide 50

BrandNiemann07182015Slide50.PNG

Slide 51

BrandNiemann07182015Slide51.PNG

Slide 52

BrandNiemann07182015Slide52.PNG

Slide 53

BrandNiemann07182015Slide53.PNG

Slide 54

BrandNiemann07182015Slide54.PNG

Slide 55

BrandNiemann07182015Slide55.PNG

Slide 56

BrandNiemann07182015Slide56.PNG

Slide 57

BrandNiemann07182015Slide57.PNG

Slide 58

BrandNiemann07182015Slide58.PNG

Slide 59

BrandNiemann07182015Slide59.PNG

Slide 60

BrandNiemann07182015Slide60.PNG

Slide 61

BrandNiemann07182015Slide61.PNG

Slide 62

BrandNiemann07182015Slide62.PNG

Slide 63

BrandNiemann07182015Slide63.PNG

Slide 64

BrandNiemann07182015Slide64.PNG

Week 6: Deployment

Decision Dashboard

Slide 65

BrandNiemann07182015Slide65.PNG

Slide 66

BrandNiemann07182015Slide66.PNG

Slide 67

BrandNiemann07182015Slide67.PNG

Slide 68

BrandNiemann07182015Slide68.PNG

Slide 69

BrandNiemann07182015Slide69.PNG

Slide 70

BrandNiemann07182015Slide70.PNG

Slide 71

BrandNiemann07182015Slide71.PNG

Slide 72

BrandNiemann07182015Slide72.PNG

Slide 73

BrandNiemann07182015Slide73.PNG

Slide 74

BrandNiemann07182015Slide74.PNG

Slide 75

BrandNiemann07182015Slide75.PNG

Slide 76

BrandNiemann07182015Slide76.PNG

Slide 77

BrandNiemann07182015Slide77.PNG

Slide 78

BrandNiemann07182015Slide78.PNG

Slide 79

BrandNiemann07182015Slide79.PNG

Slide 80

BrandNiemann07182015Slide80.PNG

Slide 81

BrandNiemann07182015Slide81.PNG

Slide 82

BrandNiemann07182015Slide82.PNG

Slide 83

BrandNiemann07182015Slide83.PNG

Slide 84

BrandNiemann07182015Slide84.PNG

Slide 85

BrandNiemann07182015Slide85.PNG

Slide 86

BrandNiemann07182015Slide86.PNG

Week 7: Project

Three Choices and Requirements

Slide 87

BrandNiemann07182015Slide87.PNG

Slide 88

BrandNiemann07182015Slide88.PNG

Slide 89

BrandNiemann07182015Slide89.PNG

Slide 90

BrandNiemann07182015Slide90.PNG

Slide 91

BrandNiemann07182015Slide91.PNG

Slide 92

BrandNiemann07182015Slide92.PNG

Slide 93

BrandNiemann07182015Slide93.PNG

Week 8: Final Exam

25 Multiple Choice Questions

Slide 94

BrandNiemann07182015Slide94.PNG

Spotfire Dashboard

For Internet Explorer Users and Those Wanting Full Screen Display Use: Web Player Get Spotfire for iPad App

Error: Embedded data could not be displayed. Use Google Chrome

USDA, Microsoft to Launch “Innovation Challenge”

USDA Press Release: Word

Top Prize of $25K Offered for Most Creative Application on Food Resiliency and Data Science

WASHINGTON, July 27, 2015 - The U.S. Department of Agriculture is partnering with Microsoft to launch the “Innovation Challenge,” a competition to develop software applications that help farmers, agriculture businesses, and consumers explore how climate change will affect their food systems.

The Innovation Challenge will be formally launched on July 27th at a conference of the Agricultural & Applied Economics Association in San Francisco. Challenge participants have 3 months to create their applications, with a top prize of $25,000 going to the most creative application that best exploits USDA data sets that are now being hosted on Microsoft Azure, Microsoft’s cloud computing platform.

Entrants are invited to develop and publish new applications and tools that can help users analyze multiple sources of information, including key USDA data sets. In addition, Microsoft is granting cloud computing awards to aid university researchers and students that are looking to take part in the challenge.  Challenge winners will be announced in December 2015.

Full details of the challenge can be found at http://usdaapps.challengepost.com. My Note: See Data Science Audit Below

Climate change will impact all aspects of the food system, from the ability to grow food, to the reliability of food transportation and food safety, to the dynamics of international trade in agricultural goods. The Food Resilience theme of the challenge seeks to inspire the creation of tools that help users analyze and explore our food systems. 

“The Innovation Challenge is an exciting outcome of the President’s Climate Change Action Plan,” said Dr. Ann Bartuska, USDA Deputy Under Secretary for Research Education and Economics. “We at USDA were asked to look at the vulnerability of the food supply to climate change, to start mining the data that we have, and to find partners like Microsoft who could help us create something new and different to make these data sets available to the larger public.”

“I am looking forward to discover what creative ideas the community comes up with through blending together USDA data,  novel ways of exploring and analyzing data with access to large scale computing and data analysis in the form of Microsoft’s Azure platform,” said Dr. Daron Green, Deputy Managing Director of Microsoft Research.

“Microsoft and the USDA seek to catalyze the creativity of others and hope that the challenge will provide a great incentive for developers and researchers interested in data science to put together some great applications helping address the USA’s food resiliency needs,” Green added.

For more than 100 years, USDA has compiled data from economic reports, satellite imagery, remote sensors, and surveys on crop production practices, financial outcomes, demographics and remote sensing products on the health of crops around the country. For the first time, key USDA datasets are being made available in the cloud where they can be accessed and blended with other data to give novel insights or used in new types of end-user applications. 

Combining these USDA data with cloud computing resources allows even very high fidelity and complex models to be processed in a timely manner and the results delivered to remote users on their laptops, tablets or mobile phones.

USDA and Microsoft Innovation Challenge for Food Resilience

YouTube Video

Source: https://www.youtube.com/watch?v=ON_wlfIwZBg

Published on Jul 24, 2015

The United States Department of Agriculture (USDA) and Microsoft are sponsoring the Innovation Challenge, a competition to develop data-driven software applications that help visualize and explore how climate change will affect the resiliency of food systems in the United States. Challenge participants will have access to a Microsoft Azure cloud-based data portal hosting key USDA datasets on agricultural practices and outcomes that span the past 100 years, enabling participants to investigate how food systems of tomorrow will be different from today.

The challenge offers prizes for applications that use the USDA data and provide actionable insights to farmers, agriculture businesses, scientists, or consumers across the United States so that they can plan for how current and future climate changes might influence food systems.

Transcript

My Note: This is just sentence fragments

0:06 farmers are inherently you wouldn't information among her
0:10 long one fireman down from generation to generation
0:17 but it seems increasingly likely young farmers
0:20 are going to see circumstances gonna our parents and grandparents
0:24 surely she
0:29 we were asses USDA to look at a vulnerability about food supply
0:34 to climate change
0:40 which meant we could start mining all the data that we had
0:43 and when we started thinking about mining data and being able to have
0:47 somebody in the private sector work with us Microsoft was there Microsoft in the
0:51 USDA are rolling out at Innovation Challenge which is really a call to
0:55 action for developers around the world
0:57 to start digging into summer the key data sets with your sister collecting
1:01 over the past 100 years
1:03 on crop production practices financial outcomes demographics and remote sensing
1:08 products so
1:09 the health of crops around the country the Innovation Challenge
1:13 I think it's just one of those exciting outcomes and this whole climate data
1:16 initiative and that is
1:18 creating public private partners like Microsoft to actually created
1:23 do something new and different and to make these
1:27 datasets available to the larger public
1:34 using data and information transferring knowledge
1:37 between people who are necessarily co located close together
1:41 gives us an opportunity to you improve our practice is based on their
1:45 understanding
1:46 know we might not otherwise have had actions solutions that answer
1:50 some of the key questions at the USTA has about how to
1:54 insurer the resiliency Americas foods that's what I think after services
1:58 brings us it's a way to
2:00 connect the dots across all these different to different sources
2:03 having a healthy farm economy benefits everyone
2:06 bringing all of those different kinds of data together
2:11 is going to be difficult it's going to be a challenge
2:14 and that's why we're asking people around the world to help the USTA Sol
2:18 this
2:23 in

Help build a sustainable US food system by putting USDA data into the hands of farmers, researchers, and consumers

Source: http://usdaapps.challengepost.com/

Being a farmer in America is not for the weak of heart. For decades, it has been a game of both chance and experience - making farming a difficult industry to enter. In fact, the number of entry-level farmers has fallen by 30% since 1987 (http://www.cfra.org). My Note: See below

For those that do enter this industry, farming is becoming an increasingly data and technology driven activity. Known as “precision farming” or “precision ag,” farmers are now utilizing data from satellites, market reports, weather forecasts, surveys, and sensors that provide on-demand GPS monitoring and mapping tools. (http://www.usatoday.comNote: See below

Still, it’s not enough.

American farmers need more data in order to create a sustainable food system for the United States. They need to analyse the food supply coming from farms and ranches and the economics of consumer demand. They need to know how yields have changed over time so they can prepare for and predict future crops. They need to know what is growing well in their area and what isn’t. Similarly, consumers and researchers want to know where their food is coming from and how we can make US agriculture more sustainable.

The USDA has a tremendous amount of food supply, economic demand, and remote sensing data as part of its Agricultural Statistical Service (NASS) and Economic Research Service (ERS), the challenge is to explore how to make this data accessible and provide insights for potential users.

Help create a sustainable, competitive, and healthy US food system. Use USDA data to create working, interactive applications to get farmers the information they need – and help feed America.

View full rules

Eligibility

This challenge is open to:

  • Individuals (who have reached the age of majority in their jurisdiction of residence at the time of entry)
  • Teams of eligible individuals
  • Organizations (up to 50 employees)
  • Large Organizations (with over 50 employees) may compete only for the non-cash Large Organization Recognition Award.
  • Employees of Large Organizations (with over 50 employees) are eligible as long as they enter the competition independent of their company and meet all other requirements.

Employees of the USDA, Microsoft, ChallengePost and contractors currently under contract work for USDA, Microsoft, or ChallengePost are not eligible. My Note: I am eligible

Requirements

What to Create: Submit a working, interactive application that integrates one or more of the required USDA datasets. 

Static data visualizations will not be eligible. Applications must include interactive functionality (e.g. the user can change parameters to update the visualization and/or result). My Note: I use interactive data visualization functionality

Eligible Platforms:

  • Smartphone or tablet (iOS, Android, Blackberry, Kindle, Windows 8 Mobile) My Note: Yes
  • Web (mobile or desktop) My Note: Yes
  • Desktop (Windows PC, Mac Desktop) My Note: Yes
  • Software running on other publicly available hardware (including, but not exclusive to, wearable technology, open source hardware, etc.) My Note: Yes

Supplemental Material: You must submit a demo video (hosted on YouTube, Vimeo, or Youku) that walks through the main functionality of the application via screencast or video. You must also submit a text description and at least one image/screenshot of your working application. My Note: Yes

Testing: You must make your app available for testing by providing a link to access your installation file, an uploaded installation file, a beta distribution build, etc. See full testing access options. My Note: Yes

New & Existing Solutions: Apps may be newly created or pre-existing. If the submitted app existed prior to the competition’s submission start date, it must have been updated to integrate the required USDA data during the submission period. My Note: Yes

Judges

Dr. Debra Peters

Senior Advisor for Earth Observations, U.S. Department of Agriculture, Office of the Chief Scientist and Research Scientist, USDA Agricultural Research Service

Dr. Debra Peters
Vince Breneman

Chief of the Research Support Branch at USDA’s Economic Research Service

Vince Breneman
Rick Mueller

Head of NASS’ Spatial Analysis Research Section in Fairfax, Virginia

Rick Mueller
Lucas Joppa

Environmental Scientist, Microsoft

Lucas Joppa
Daron Green

Deputy Managing Director, Microsoft Research Outreach

Daron Green
Dr. Brian Lutz

Third Generation Farmer, Ray Lutz Farms

Dr. Brian Lutz

Judging Criteria

  • Quality of Idea 
    Includes creativity and originality of the idea.
  • Implementation of Idea
    Includes the quality of the design and user experience, as well as the level of analysis difficulty of the required data.
  • Clarity and Accuracy of Solution
    Includes the completeness of the documentation and the accuracy of the data usage.
  • Potential Impact
    Includes the extent to which the application could be useful to researchers, agricultural workers, etc

View All Dates

Source: http://usdaapps.challengepost.com/details/dates

 

PERIOD BEGINS ENDS
Submissions July 24 at 10:00 am EDT October 27 at 5:00 pm EDT
Public Voting November 04 at 10:00 am EST November 17 at 5:00 pm EST
Judging November 04 at 10:00 am EST November 17 at 5:00 pm EST
Winners Announced   December 02 at 2:00 pm EST

Prizes

$63,000 in prizes
 Grand Prize
USD$25,000 cash prize
 Second Prize
USD$15,000 cash prize
 Third Prize
USD$7,500 cash prize
 Honorable Mention (2)
USD$2,000 cash prize
 Open Source Applications (3)
USD$2,500 cash prize. To be eligible for the Best Open Source Application Award, the submitter must make their application code available to the public at the time of entry under one of the approved open source licenses listed at: http://opensource.org/licenses. See rules for full requirements.
 Best Visualization in Time or Space Award
USD$2,000 cash prize. To be eligible for the Best Visualization in Time or Space Award, a Submission must meet the requirements of one of the following categories: Data Visualization of Required Data Over Time, Map Overlay, App Including Crowdsourced Data. See rules for full requirements.
 Best Student-made App
USD$1,500 cash prize. Awarded to the best app created by a one or more student hackers. (All team members must be taking classes at an post-secondary educational institution to qualify, or must have graduated within the last 90 days. See rules for full requirements. Large organizations are not eligible for the Best Student-made App.)
 Popular Choice Award
USD$500 cash prize.
 Large Organization Recognition Award
Non-cash, recognition only for entrants with more than 50 employees.

How To Enter

  1. Click “Register” to sign up for important challenge communications. My Note: Decide on this
  2. Visit the Resources page for a list of the eligible USDA datasets to get you started. My Note: See Below
  3. Create your app! My Note: Done two apps already
  4. Shoot your demo video and take screenshots of your functioning app. My Note: This will be done as part of my online course
  5. Provide a way for us to access your app. My Note: See Below
  6. Get started on your draft and submit early! My Note: I started before this was announced

Beginning Farmer and Rancher Opportunities

Source: http://www.cfra.org/resources/beginning_farmer

My Note: Excellent page

Technology is king down on the farm

Source: http://www.usatoday.com/story/money/...rming/1741071/

My Note: December 2, 2012 article

Rules

My Note: I skim read this

Video should be less than five (5) minutes

C. Best Visualization in Time or Space Award Category Requirements

To be eligible for the Best Visualization in Time or Space Award, a Submission must meet all of the Submission Requirements described in Section 4, and the requirements of one of the following sub-categories:

(i) Best Data Visualization of Required Data Over Time

The Application includes a visualization tool that allows users to view either: (a) trends through time for multiple variables for the same location, or (b) the same variable(s) across a spatial extent (the entire US, or a region). Makers are encouraged to combine the USDA data with other data.

(ii) Best Map Overlay

The NASS cropscape, vegscape, and agriculture census maps (and underlying data), and ERS atlases (maps with data) are all mapped at the county level. The Application includes map overlays at the county or state level, and conducts analyses of these map overlays. For example, the analysis could show the number of pixels of corn planted on different soil types in Nebraska, or the distribution of distance between cornfields and bioenergy plants in the Midwest.

(iii) Best App Including Crowdsourced Data

The Application includes a real-time component by integrating the ability to capture and integrate crowdsourced data in addition to USDA data.

Winners of the Best Visualization in Time or Space Award are eligible to win other prizes described in Section 7, provided they meet any applicable Prize-specific eligibility and category requirements. Such winners are not eligible for the Large Organization Recognition Award, and Large Organizations are not eligible for the Best Visualization in Time or Space Award.

My Note: Unfortunately, I think I fail this criteria because I did my work with the USDA data sets before July 24th.

(iv) New & Existing: Applications must be either newly created by the Maker or, if the Application existed prior to the Challenge Submission Period, must have integrated the required USDA data after the start of the Challenge Submission Period.

Submissions

Nothing to see here yet. The organizers have not made submissions public. Check back soon to see awesome software.

Updates

Stay tuned for important announcements. Organizers will post them here and also send an email to registered participants.

Discussions

My Note: Can request Email me when new discussions are added

Sibling works for USDA, am i disqualified? No

Resources & Data

Required USDA Datasets

To participate in this challenge you must create and submit a working, interactive application that integrates one or more of the required USDA datasets. (Please review the official rules for full details.) 

USDA datasets include information gathered from NASS and ERS and include assets such as:

  • NASS Quickstats API – The National Agricultural Statistics Service (NASS) provides direct access to a database containing official published aggregate estimates related to U.S. agricultural production. It includes hundreds of sample surveys covering virtually every aspect of U.S. agriculture and providing state and county-level aggregates. APIs are available for this dataset. My Note: Used the entire data set and not the API
  • ERS ARMS Farm Financial and Crop Production Practices - The field-level phase collects information on production practices and costs (such as fertilizer, pesticide, labor, tillage, and seed) for target commodities. The farm-level phase collects financial information for farm businesses and a variety of financial and demographic information (such as age, education, occupation, off-farm income) for farm operators and their households. The survey collects information from 48 states. It is designed to provide representative data of the continental United States and to support state-level estimates for 15 key agricultural states. APIs are available for this dataset. My Note: I hope more than APIs are available to use this data. I found the data download
  • NASS CropScape API The National Agricultural Statistics Service (NASS) offers a CropScape API that provides direct access to a raster image dataset containing an agricultural-specific land cover classification published once annually at the end of growing season. Data is only offered at original data resolution of 30 to 56m. APIs are available for this dataset. My Note: I hope more than APIs are available to use this data. Not sure how a farmer or data scientist uses raster data.
  • NASS VegScape API - The National Agricultural Statistics Service (NASS) offers the VegScape API that provides direct access to a raster image dataset containing the crop condition vegetation NDVI, published on biweekly timescales throughout the year. Original data resolution is 250m but data can be retrieved at any resolution desired by the user. APIs are available for this dataset. My Note: I hope more than APIs are available to use this data. Not sure how a farmer or data scientist uses raster data.

To easily access the data via Azure, visit the Farm Data Dashboard 

The Farm Data Dashboard is a Microsoft Azure reporting service that offers a simple interface providing an instant overview across several complex datasets on farms and crops. Microsoft’s Farm Data Dashboard is one of several ways that Microsoft cloud products and machine learning capabilities are making the USDA’s data accessible to external developers to help them build applications and services that strengthen and streamline the nation’s food systems.

My Note: This link goes to 

NASS (Quickstats)

Select one or more items to filter records. There are currently 31,613,259 records available.

My Note: This is the same interface to Quickstats as the USDA Web Site and that I did in Spotfire.

ARMS

Select one or more items to filter records. There are currently 12,140,959 records available.

My Note: Is this entire data set available to download? Yes, it is and I downloaded it!

CropScape

My Note: This opens to a blank screen

VegScape

My Note: This opens to a blank screen

Use cases

My Note: I looked at these do not see why I would use them

AgMIP

My Note: This page says I can download Metadata (Zip files) for Maize, Soybean and Bread Wheat

Documentation

My Note: These are the same 5 PDF files shown below

Download

My Note: Here is where I can download the following:

Databases (2)

Nass_database.zip (623 MB) Already downloaded and used this in my USDA Data Science MOOC

Arms_database.zip (511 MB) I download this in 2 minutes!

Cropscape (8 multiple GB tif files)

VegScape (many 2010-2015 0.5 GB files)

AgMIP (many ACEDB and ACMO zip files for Maize, Soybean, and Bread wheat for selected states, but not Nebraska)

Support & Documentation

Documentation for each dataset can be found at http://innovationchallenge.azurewebsites.net/#DocumentationTabIf you have questions about the data, the competition, etc. please post them to the competition discussion board.

My Note: This link goes to http://innovationchallenge.azurewebsites.net/#NassTab

It should go to: http://innovationchallenge.azurewebs...cumentationTab

where there are 5 PDF files that I find difficult to use:

http://challengefiles2.blob.core.win...escription.pdf

http://challengefiles2.blob.core.win...escription.pdf PDF See Excel Tab: Table Column Definitions

http://challengefiles2.blob.core.win...escription.pdf

http://challengefiles2.blob.core.win...escription.pdf

http://challengefiles2.blob.core.win...escription.pdf

Additional Resources

Other Websites that reference the CDL/CropScape:

Visit the Plan East Tennessee project: ET Index Web Map Application

Visit the Texas A&M AgriLife Research at Beaumont: iAMS Cropland Data

Visit a Bioenergy/CDL web mashup Bioenergy Knowledge Discovery Framework/U.S. Dept. of Energy My Note: Is this the tie in to the Best Map Overlay suggestion below?

Getting Started with an Idea

Interested in the challenge, but not sure how to start? Try thinking about how you can help farmers, researchers, and consumers answer questions such as:

My Note: These are good questions to answer

  • How much food do we create every year and how is it distributed?
  • Where is food produced and what is the process by which consumers are fed?
  • How have our food preferences changed through time?
  • Where does our food come from?
  • What varieties of crops are being planted and what is doing well in different areas?
  • Is there a disparity between production and purchasing in local growing environments?

Requirements for Bonus Categories

Best Student-made App Award

The Best Student-made App Award will be awarded to the best app created by one or more student developers. All team members must meet the basic requirements in Section 3 of the rules and:

  • be currently enrolled in at least nine credits or three courses, or the equivalent at the time of entry (or must have been enrolled in such credits or courses within the past three months); OR
  • have graduated in the three months prior to the date of entry from either a secondary school or functional equivalent, or an accredited post-secondary institution (e.g., university, community college, technical college).

Winners of the Best Student-made App Award are eligible to win other prizes, provided they meet any applicable Prize-specific eligibility and category requirements.

Best Visualization in Time or Space Award

To be eligible for the Best Visualization in Time or Space Award, a submission must meet all of the Submission Requirements described in Section 4 of the rules, and the requirements of one of the following sub-categories:

  1. Best Data Visualization of Required Data Over Time - The Application includes a visualization tool that allows users to view either: (a) trends through time for multiple variables for the same location, or (b) the same variable(s) across a spatial extent (the entire US, or a region). Makers are encouraged to combine the USDA data with other data.
  2. Best Map Overlay - The NASS cropscape and vegscape are mapped at the county level. The Application includes map overlays at the county or state level, and conducts analyses of these map overlays. For example, the analysis could show the number of pixels of corn planted on different soil types in Nebraska, or the distribution of distance between cornfields and bioenergy plants in the Midwest.
  3. Best App Including Crowdsourced Data - The Application includes a real-time component by integrating the ability to capture and integrate crowdsourced data in addition to USDA data.
Best Open Source Application Award

To be eligible for the Best Open Source Application Award, you must make your submitted application code available to the public at the time of entry under one of the approved open source licenses listed at: http://opensource.org/licenses. My Note: I am not using Open Source and not doing Coding for my data science analytics and visualization work.

More Questions?

For questions about USDA Innovation Challenge, email Support@ChallengePost.com, or post them to the DiscussionBoard.

FAQ

My Note: I skim read this

Story

USDA Big Data Science for Precision Farming With FarmLogs

Working the Land and the Data Schematic.jpg

Email to the FarmLogs Customer Success Team on June 26, 2015:

I am a Ph.D. data scientist/data journalist and organizer of the Federal Big Data Working Group Meetup in Washington, DC.

We have done two very well-received Meetups with the USDA on their Open Data Plan (March 16th) and a USDA Data Science MOOC for Data Science Training (May 18th).

Joyce Hunter, USDA CIO, said at our first Meetup, they would like help with a MOOC, which we did, and Precision Farming, which we now planed for September 14th.

The NYT article and the Fox Business News report brought you to our attention.

I would like to invite you to participate in our September 14th Meetup, which I am tentatively titling:

USDA Big Data Science for Precision Farming With FarmLogs

I am trying to capture that we are continuing to use Data Science on USDA data sources and how you have applied those data sources in an application that 20% of the farms use now for precision farming.

I would like to collaborate with you and the USDA on this Meetup so our 800+ members can understand how you used the Data Science Data Mining Standard to get to your application and new business:

  1. Business Understanding
  2. Data Understanding
  3. Data Preparation
  4. Modeling
  5. Evaluation
  6. Deployment

I also would like to answer the 4 Data Science Questions we want answered in our presentations:

  • How was the data collected?
  • Where is it stored?
  • What are the results?
  • Why should we believe the results?

I look forward to hearing of your interest in preparing for and presenting at this future meetup. We provide for remote presentations if travel is a problem.

For the 1. Business Understanding and 2. Data Understanding steps, I start with excerpts of NYT article to investigate further:

Four years later, thousands of farmers worldwide use their app, FarmLogs, to record planting dates, watering schedules and crop yield. In addition, subscribers can receive data from FarmLogs about rainfall and soil health that is tailored to their fields.

The FarmLogs app allows them to enter the information on mobile devices and to share it easily via the web. Certain bits of information, such as rainfall accumulation and crop health, are gathered automatically by the FarmLogs app, saving farmers time and miles.

This spring, the three-year-old company introduced ways to remotely monitor soil health and crop yield.

In fact, most of FarmLogs is free for now, with the exception of the Automatic Activity Recording package, a $300-a-year offering in which the app detects when certain work is taking place in the field and logs information without the user needing to input it manually.

Mr. Vollmar said he quickly realized that farmers did not only want a more convenient way to handle and gain access to the data they generated. They also wanted a supply of easy and digestible information about their farms that would help improve their yields. By summer 2013, FarmLogs began offering rainfall data collected for each square kilometer of farmland in the nation by the National Weather Service, information that could tell farmers more precisely where rain had fallen on their land and approximately how much.

That data would soon be augmented by soil health data collected by the Department of Agriculture and, more recently, proprietary satellite imagery with spectrometry that can assess plant health.

“When you sign up to FarmLogs today, you get 10 years of rainfall data, 10 years of thermal conditions on your field and now five years’ worth of history on these fields from the satellite to build up a baseline,” Mr. Vollmar said. “We monitor that field all season long, and whenever there’s a problem, we alert the farmers and pinpoint the exact spot on the field that needs their attentions.”

Not everyone is a fan. An agriculture economics professor, Steve Ford of Sewanee University in Tennessee, is skeptical about FarmLogs and the entire crop of emerging apps of its sort. He contends that the company overstates the number of active users as opposed to casual ones and that his experience showed that FarmLogs’ rainfall data is, at best, imprecise. The app, even with all of its information sources, can’t know many variables, such as field irrigation infrastructure and fertilizer and herbicide choices.

An earlier version of this article misstated how the FarmLogs app gathers information and what sort of information it gathers. It gathers certain information, like rainfall accumulation and crop health, automatically; it does not use sensors in the field to transmit data on the times and amounts of watering. The earlier version also misstated the amount of initial data the company offers customers. The data includes 10 years of rainfall data and 10 years of thermal conditions, not 20 years of each.

Next I want to find data so I can do 3. Data Preparation, 4. Modeling, and 5. Evaluation, so then I can 6. Deployment of my own Data Science for Precision Farming. A lofty goal for the next 2-3 months before the Meetup on September 14th!

In my exploration for "precision farming data", I found a USDA report: Rice farms are adopting precision agriculture technologies

Ricefarmsareadoptingprecisionagriculture2006and2013.png

This chart is found in the joint ERS/National Agricultural Statistics Service (NASS) report, 2013 ARMS—Rice Industry Highlights, based on ARMS Farm Financial and Crop Production Practices data. 

The reports states:

Precision agriculture refers to a set of practices used to manage fields by measuring variations in nutrient needs, soil qualities, and pest pressures. In 2013, USDA conducted the latest Agricultural Resource Management Survey (ARMS) of the U.S. rice industry, interviewing farmers about production practices, resource use, and finances in the 10 largest rice-producing States. Some technologies have been rapidly adopted; in particular, yield monitoring increased in use to 60 percent of farms between 2006 and 2013. Monitors can identify variations in yields within a field, allowing farmers to adjust inputs and practices accordingly. Auto-steer or guidance systems are now used on over half of all rice farms; these reduce stress on operators, and reduce errors in input application overlaps and seeding cut-off at the end rows. The cost savings from using these two technologies can also be accompanied by increases in yields.

So the USDA has been analyzing precision agriculture, at least for rice between 2006 and 2013, and found an increase in rice farms using it in all 5 categories. My question then is, has this been done by the USDA for other crops and are FarmLogs claim of thousands of users in 4 years proof for the farmer-crop users they serve? I did not find any statistics on their web site. I am also amazed that the USDA decided to pull-out this excerpt from the Rice Industry Highlights Reports PDF file so it could be more ready found from a Web search. This shows the value of rendering PDF files in HTML format!

The next step seems to be to list the data sources and output features that FarmLogs provides to see if they have real value to precision farming as follows:

  • Standard Features
    • The status of your farm right on the map
      • Stay organized with the big picture of your farm this season. See your fields, crops, and even nearby elevator pricing — all on our beautiful satellite maps.
    • See rain at the field
      • Check your fields' rainfall right in FarmLogs. We already have over 10 years of rainfall history just waiting for you to take a look.
    • Watch heat accumulation build your crop
      • Get a day-by-day look at how many heat units are accumulating at the field and compare to prior seasons.
    • See what your fields are made of
      • Make more informed decisions about crop management, nutrient management, and soil and water conservation with the help of field-specific soil maps.
    • Import your yield data files for mapping and analysis
      • See your yield data displayed as a color-coded overlay on the map for each of your fields. Access your yield maps to study field performance at high precision from both the web browser and FarmLogs mobile apps.
    • Log notes and activities from the field
      • FarmLogs works where you work. Capture information in the moment so nothing is ever lost. Your information is automatically synced so everyone on your farm can see what they need to see.
    • Easily plan your inputs for the season
      • No more messy spreadsheets! FarmLogs' input planning tool allows you to view your crops, customize your inputs and rates, and see your budget in just two minutes.
    • Simple inventory management
      • Monitor the movement of your grain with ease this harvest season. See what's stored where and view a transaction history as you move grain from the field, into storage, and to the buyer.
    • Growth Stage Analysis COMING SOON
      • Drive less and know more using the most advanced in-season growth stage estimates on the market. Whether your crops are still in their early growth stage or quickly approaching maturity, FarmLogs Growth Stage Analysis can keep you more informed than ever before. Reacting to yield threats and addressing limiting factors happens faster by knowing which growth stage your plants are in. Planning for harvest is a breeze with continuously-updating field-level growth stage and maturity estimates.
        • Most growth stage estimates are made using relative maturity of the seed that you plant and the measurement of growing degree days (GDD) since planting. This approach is valid, but it only tells a portion of the story. FarmLogs combines this method with our field-level Wide Dynamic Range Vegetation Index observations to determine a much more precise and accurate growth stage estimate than models that use GDD alone. Satellite observations taken of each field multiple times throughout the growing season allow us to calibrate predictions to reflect ground truth conditions. Simply enter a planting date and FarmLogs will keep you informed.
  • Advantage Features
    • Automatic Activity Recording
      • Harness the power of your smartphone's GPS to automatically record all of the activities happening on your farm. By simply installing the FarmLogs mobile app, you can rest assured that field operations will be automatically recorded. Now you can keep more detailed and accurate records without having to spend additional time entering information manually.
        • Subscription: $299/feature user/calendar year
          • Automatic activity recording works by using powerful geofencing algorithms developed by FarmLogs to detect when work is being performed out in the field. The mobile app automatically recognizes when it enters a field and begins gathering information to intelligently determine details about an activity. Push notifications collect additional information from the operator, if necessary, to ensure that no information is lost or forgotten. Simply install the FarmLogs mobile app on your Android or iOS device and let FarmLogs do the work for you.
    • In-Season Crop Health Monitoring
      • Monitoring and scouting fields has never been this easy. Building upon our proprietary Field Performance Maps, FarmLogs Crop Health Monitoring is able to detect unusual behavior in the field during the growing season. Automatically receive alerts when your crop's health starts to look different from normal at a given point in the growing season. Find and fix problems at the field before they hurt your yield. Cover more ground scouting by prioritizing the fields that need the most attention. Let our mobile app guide you to the exact location in the field that needs to be scouted.
        • This feature is FREE for 2015
          • Five years of historical measurements are used to create performance baselines for every 5 square meters of your fields. The health of the crop is then monitored in-season against this baseline to detect anomalies. When the crop looks different at a location in your field from how it normally would, FarmLogs creates an alert for that field and shows the exact location of the area that needs to be investigated.

So let's consult the FarmLogs Support to find the following for Fairfax County, Virginia:

  • Satellite maps: Between May 15 and September 15, a satellite passes over your fields approximately every 5 days. As long as the satellite is able to capture an image that is at least 75% clear of cloud cover, we process that image and add it to your account. Most images appear in your account within 48 hours of being captured by the satellite.
  • Shape file: Requested
  • Rainfall data: There is a known issue with delays in rainfall data from the National Weather Service. Our rainfall history and tracking comes from a dataset that the National Weather Service produces. The National Weather Service uses multiple radar and ground stations to algorithmically calculate the amount of precipitation that falls on a high-resolution grid across the entire United States. The National Weather Service factors in variables like wind and terrain that influence where the rain actually hits the ground. This is done within 1.2 miles of accuracy. We then check their rainfall grid data for updates every hour and match it up with your field boundaries.
  • Heat accumulation:
  • Soil maps: FarmLogs is investigation better ways to track soils testing
  • Yield data: See Which yield files can FarmLogs import?
  • Crop health: See above Advantage Feature
  • Growth stage: We developed a new growth stage model to provide more accurate growth stage estimates of your fields. Most models estimate growth stage based solely on growing degree days (GDD). Our model uses actual observations of your fields, extracted from satellite imagery, in conjunction with GDD to estimate the current growth stage of your fields. This results in more precise growth stage estimates.
  • Growing Degree Days: FarmLogs uses the USDA's formula for calculating corn growing degree days (GDD). This is an 86/50-bounded calculation, which only allows GDD to accumulate when temperatures are optimal for corn growth. This means that we take the average temperature of the day (bounded by a minimum of 50° and a maximum of 86°) and then subtract 50 to provide you the average number of degrees above 50° that the crop accumulated.
  • Shrinkage: We automatically shrink the wet amount that you enter to 14%. We display both the wet and dry amounts with the shrinkage rate applied. It is not currently possible to adjust the shrinkage rate, but this is an enhancement that we are considering for a future version.
  • Etc.

MORE TO FOLLOW

Data Mine FarmLogs My Note: Started

Data Mine Google Precision Farming: e.g. Rice farms are adopting precision agriculture technologies My Note: See above

Understand the USDA Data Sets and Models Used in Precision farming

The Use of Farm Field Sensors

Etc.

Slides

Slides

Slide 1 USDA Big Data Science for Precision Farming With FarmLogs

Semantic Community

Data Science

USDA Big Data Science for Precision Farming With FarmLogs

BrandNiemann09142015Slide1.PNG

Slide 2

BrandNiemann09142015Slide2.PNG

Slide 3

BrandNiemann09142015Slide3.PNG

Slide 4

BrandNiemann09142015Slide4.PNG

Slide 5

BrandNiemann09142015Slide5.PNG

Slide 6

BrandNiemann09142015Slide6.PNG

Slide 7

BrandNiemann09142015Slide7.PNG

Slide 8

BrandNiemann09142015Slide8.PNG

Slide 9

BrandNiemann09142015Slide9.PNG

Slide 10

BrandNiemann09142015Slide10.PNG

Slide 11

BrandNiemann09142015Slide11.PNG

Slide 12

BrandNiemann09142015Slide12.PNG

Slide 13

BrandNiemann09142015Slide13.PNG

Slide 14

BrandNiemann09142015Slide14.PNG

Slide 16

BrandNiemann09142015Slide16.PNG

Research Notes

Upload Shapefiles

https://app.farmlogs.com/#fields/upload

Shapefiles are computer files that contain the boundaries of your fields. You can upload them here to import your field boundaries into FarmLogs.

Request your shapefiles from the FSA

If you already have your shapefiles, skip to the next step.

You can look up the email address for your county FSA office's executive director here and send them an email requesting your shapefiles. My Note: I did this

Show sample email

To whom it may concern,

I would like to request digital "Shapefile" copies of the Common Land Units (CLUs) associated with my production in accordance with the rules and procedures set forth under FSA Handbook 8-CM, Par. 253 C.

I look forward to your prompt return of this data.

Regards,

Brand Niemann

Google: fairfax county gis shapefiles for agriculture

Agricultural Data for Fairfax County
http://www.fairfaxcounty.gov/demogrp...d_by_topic.htm

Data and Statistics

http://www.nass.usda.gov/Data_and_Statistics/

My Note: Last modified: 06/30/10 Out of Date?

Future Vision for the Use of Remotely Sensed Data

http://www.nass.usda.gov/Surveys/Rem...sion/index.asp

Last modified: 05/27/09

Organic Farming Survey

http://www.agcensus.usda.gov/Publica...rganic_Survey/

2012 Census Publications
http://www.agcensus.usda.gov/Publica...iles/index.php

Download all the data in CSV format
http://www.agcensus.usda.gov/Publica...ofiles/cdp.csv

My Note: Not useful

FarmLogs App Helps Farmers Collect and Retrieve Data

Source: http://www.nytimes.com/2015/06/25/bu...ieve-data.html

By STEVE FRIESS

JUNE 24, 2015

 

Jesse Vollmar, left, and Brad Koch both come from families with farming backgrounds. Credit Tim Galloway for The New York Times

ANN ARBOR, Mich. — One day in 2011, Jesse Vollmar’s uncle was complaining about how difficult the family farm’s new data management software was to use. Despite taking a course taught by a software provider, he still couldn’t figure out how to get it to keep track of seeding dates, rainfall, field yield or any of the other functions that mattered most.

Mr. Vollmar, now 26, thought that he and his partner in a local website design business could do better. Four years later, thousands of farmers worldwide use their app, FarmLogs, to record planting dates, watering schedules and crop yield. In addition, subscribers can receive data from FarmLogs about rainfall and soil health that is tailored to their fields.

It is a striking departure from just a few years ago, when farmers had to travel to their far-flung fields, scribble such data in notebooks and input it into hard-to-use software that resided on a specific desktop computer. The FarmLogs app allows them to enter the information on mobile devices and to share it easily via the web. Certain bits of information, such as rainfall accumulation and crop health, are gathered automatically by the FarmLogs app, saving farmers time and miles.

TECHNOLOGY By Michael Kirby Smith and Ben Laffin 5:08

Bushels and Bytes: The Data-Driven Farm

The Toms in Indiana have seen farming advance from horse-drawn plows to self-guiding tractors. They say innovation has been the key to their success. By Michael Kirby Smith and Ben Laffin on Publish Date November 30, 2014. Photo by Michael Kirby Smith for The New York Times.

“In 2011, the Internet had changed the world but hadn’t yet changed farming,” Mr. Vollmar said recently, clad in a gray hoodie and sneakers as he showed off the company’s second-story office of weathered hardwood and exposed pipe that overlooks the Ann Arbor farmers’ market but would fit neatly in Silicon Valley.

As of this spring, Mr. Vollmar said, some 70,000 row-crop farms of 100 acres or more in the United States are using some elements of the FarmLogs software to keep tabs on vital information of the growing season. This spring, the three-year-old company introduced ways to remotely monitor soil health and crop yield.

The app compares real-time satellite images of every five square meters of field with the last five years of satellite imagery to detect whether a particular area is distressed. If so, farmers receive push notifications urging them to go to that spot and see for themselves why the plants are struggling compared with previous years. This option is available free this year; it is likely to require paid subscriptions in 2016, Mr. Vollmar said.

In fact, most of FarmLogs is free for now, with the exception of the Automatic Activity Recording package, a $300-a-year offering in which the app detects when certain work is taking place in the field and logs information without the user needing to input it manually.

Michael Morris, whose family farms 2,600 acres of corn, wheat and soy in Deerfield, Mo., last year tried several apps but said only FarmLogs could handle his volume of data. “We can do better nutrient placement decisions throughout the year with this technology,” he said. “No other program really offers that.”

Like the leaders of many tech start-ups, Mr. Vollmar, the chief executive of FarmLogs, speaks in lofty terms of the change-the-world importance of this technology.

“The problem we’re solving is that the world needs to feed nine billion people by 2050, and we’re not going to get any more farmland,” he said. “We can help get more out of our natural resources, make our land more productive, lower the cost of food production and build a healthy viable technology business at the same time.”

Mr. Vollmar, a fifth-generation soy and corn farmer near Saginaw, Mich., spent many summers hoeing fields and driving tractors. But his passions lay in exploring new technology. In 2004, while in high school, he began selling fellow students space on a computer server to upload their school papers so they could access the files from any web-connected computer, an early and localized version of what’s now known as the cloud. By his senior year, he and Brad Koch, a friend, started building websites for local companies in Michigan’s rural Thumb region, an enterprise so lucrative they put themselves through college at Saginaw Valley State University with its proceeds.

Mr. Koch, 25, FarmLogs’ chief technology officer, came from a family two generations removed from agriculture but was, nonetheless, surrounded by farms.

“We grew up around all these people, we knew how this business worked and we saw this is an opportunity to work on cool technology that could make a really, really big impact on all those lives,” said Mr. Koch, whose father is a chemical engineer at Dow Chemical in Midland, Mich. “We just understand what the farmers need better than someone sitting in an office in Silicon Valley.”

The pair began tinkering with early ideas for FarmLogs in 2011 and won a berth in a group of start-ups to receive seed money, technology advice and living quarters for three months in Mountain View, Calif., from Y Combinator in 2012. Alumni from earlier Y Combinator groups include Dropbox, Airbnb and Reddit.

After the Y Combinator stint, Mr. Vollmar and Mr. Koch moved their company to Ann Arbor to be closer to the agriculture industry and to tap the University of Michigan’s vaunted computer science programs for employees. By fall 2012, FarmLogs had opened for business as a web-based data management tool and drew $1 million in seed funding from the Chicago-based Hyde Park Venture Partners. In 2014, the business received an additional $14 million from Hyde Park and other sources.

Mr. Vollmar said he quickly realized that farmers did not only want a more convenient way to handle and gain access to the data they generated. They also wanted a supply of easy and digestible information about their farms that would help improve their yields. By summer 2013, FarmLogs began offering rainfall data collected for each square kilometer of farmland in the nation by the National Weather Service, information that could tell farmers more precisely where rain had fallen on their land and approximately how much.

That data would soon be augmented by soil health data collected by the Department of Agriculture and, more recently, proprietary satellite imagery with spectrometry that can assess plant health.

“When you sign up to FarmLogs today, you get 10 years of rainfall data, 10 years of thermal conditions on your field and now five years’ worth of history on these fields from the satellite to build up a baseline,” Mr. Vollmar said. “We monitor that field all season long, and whenever there’s a problem, we alert the farmers and pinpoint the exact spot on the field that needs their attentions.”

Not everyone is a fan. An agriculture economics professor, Steve Ford of Sewanee University in Tennessee, is skeptical about FarmLogs and the entire crop of emerging apps of its sort. He contends that the company overstates the number of active users as opposed to casual ones and that his experience showed that FarmLogs’ rainfall data is, at best, imprecise. The app, even with all of its information sources, can’t know many variables, such as field irrigation infrastructure and fertilizer and herbicide choices.

“I don’t have a problem with compiling information, but I don’t do it unless there is going to be some management payoff that makes it worthwhile,” Dr. Ford said. “That seems to be what is lacking here.”

Mr. Vollmar was dismissive of such criticism. FarmLogs is growing rapidly, having increased its staff to 32 full-time employees now from about 15 last fall, he said, and will have 60 by year’s end. He noted that providing free access to many tools and inviting varying levels of engagement is common among technology companies. If the software is useful, he predicted, farmers will subscribe in coming years.

“There’s always going to be people who love to pick on new technology,” Mr. Vollmar said. “Any time you’re doing something new, people challenge it and find little nitty-gritty details to pick on. We try to be transparent to farmers about what the data is and how accurate we think it is. And they seem to be happy with it so far.”

Farmers Flying Drones May Soon Be Given Clearance

MAY 15, 2015

Jean Hediger, a Colorado farmer, using a drone to pinpoint which parts of her fields need fertilizer, water, weed killer or seed.

My Note: This sentence was not on the visible page!

 

 

Jimmy Underhill, a drone specialist at Agribotix, working on a customer's drone in Boulder, Colo. The company has sold about 100 devices to farmers and crop consultants.CreditMichael Ciaglo for The New York Times

Working the Land and the Data

NOV. 30, 2014

Working the Land and the Data Schematic.jpg

A diagram drawn by Kip Tom shows the workings and various parts of the Tom Farms operations. The drones and satelittes are still under consideration. Michael Kirby Smith for The New York Times

Correction: June 24, 2015 

An earlier version of this article misstated how the FarmLogs app gathers information and what sort of information it gathers. It gathers certain information, like rainfall accumulation and crop health, automatically; it does not use sensors in the field to transmit data on the times and amounts of watering. The earlier version also misstated the amount of initial data the company offers customers. The data includes 10 years of rainfall data and 10 years of thermal conditions, not 20 years of each. The earlier version also misstated Brad Koch’s title. He is chief technology officer, not chief operations officer. And, because of an editing error, the earlier version misstated how much additional funding the business received in 2014. It was $14 million, not $15 million.

Note: June 25, 2015

A version of this article appears in print on June 25, 2015, on page B5 of the New York edition with the headline: Building an App That Lets Farmers Record, Track and Retrieve Data . Order Reprints| Today's Paper|Subscribe

Google didn’t lead the self-driving vehicle revolution

Source: https://www.washingtonpost.com/blogs...ohn-deere-did/

John Deere did.

By Andrea Peterson June 22

Video Link: John Deere demonstrates its AutoTrac™ guidance, which allows the tractors to self-steer and follow pre-programmed routes. (John Deere)

Google has received tons of gushy press for its bubble­shaped self­driving car, though it's still years from the showroom floor. But for years John Deere has been selling tractors that practically drive themselves for use on farms in America's heartland, where there are few pesky pedestrians or federal rules to get in the way.

For a glimpse at the future, meet Jason Poole, a 34­year­old crop consultant from Kansas. After a long day of meetings earlier this month and driving five hours across the state to watch his little girl's softball game, he was still able to run his John Deere tractor until 2 a.m. thanks to technology that left most of the driving up to a computer.

The land is hilly on Poole's family farm, so he drives the first curved row manually to teach the layout to his tractor's guidance system and handles the turns himself. But after that, he takes his hands off the steering wheel and allows the tractor to finish.

"We kind of laugh when we see news stories about self­driving cars, because we've had that for years," Poole said.

And the advancements being rolled out on the farm could soon show up next door: Your neighbor can already replace his lawn mower with the John Deere equivalent of a Roomba robotic vacuum for his yard.

The self­driving technology being sold by John Deere and some of its competitors are less technically complex than the fully driverless cars that big tech companies and car manufacturers are working on. And for now, the tractors are still supposed to have a driver behind the wheel ­ even if they never touch it.

But they've already started to transform farming in America and abroad: John Deere is selling auto­steering and other self­guidance tech in more than 100 countries, said Cory Reed, vice president of the company's Intelligent Solutions Group.

"John Deere is the largest operator of autonomous vehicles," said Catherine J.K. Sandoval, a California public utilities commissioner at a recent event hosted by the Federal Trade Commission.

Video Link: Tractor-makers such as John Deere have been selling systems that let tractors practically drive themselves. (Greg Peterson)

Some farmers aren't shy about their enthusiasm for the tech ­ even uploading videos showing it off online. One Mobility and Technology Apple Car rumors are revving up again Verizon’s Internet TV app will drop in a matter of week appears to show a tractor hauling a planter making a tightly choreographed turn without a driver in the cab. In another, the driver takes pictures, throws paper airplanes and balances a water bottle on his nose before appearing to nod off while the tractor keeps working his field.

The systems come with their own risks, including concerns that they could be hacked. But because farm­equipment makers operate almost exclusively on private land, they've been able to bring products to market much quicker than consumer automakers ­ and without the same level of regulatory scrutiny.

There are no federal rules specifically addressing self­driving tech for tractors, largely because farm equipment is designed for use in fields where it doesn't pose the same level of risk to other vehicles or people as a self­driving vehicle on a public road. The closest thing to national regulations are safety standards set by the Occupational Safety and Health Administration, but the agency does not have any rules directly aimed at self­driving technology.

That lack of regulations is one reason that the future reached the farm first. But another is pure necessity: There's a labor crunch in rural America ­ young people move to the cities, leaving the average age of U.S. farmers at 58, according to the Department of Agriculture. Similar forces are pushing self­driving tech into other industrial sectors at a pace that outstrips the consumer market. Earlier this year, the first self­driving semi­truck licensed to drive on public roads in the United States made its debut in Nevada with a splashy press show. And self­driving trucks are already being rolled out for mining and oil operations in remote parts of the world.

The success of self­driving tech in agriculture could serve as a guiding post for carmakers who are now trying to figure out how to move similar technologies onto U.S. streets.

Tractor makers started exploring the possibility of putting software in the driver's seat decades ago. At first, they used satellite technology to help farmers plot courses that they could drive manually. About 15 years ago John Deere started using similar systems to guide the tractors automatically, Reed said.

Crops are generally planted in one long row after another ­ and farmers want to make sure they use up all their land. Before auto­steering, that often meant a few feet would get reworked in every row. But with modern technology, that overlap can be reduced to less than an inch so it takes fewer passes to cover each field ­ saving farmers time and money.

Tractor­-buyers today can pile on modular systems that give computers more and more control over operations. You start with a basic satellite guidance system and a touchscreen interface, but from there the options are almost endless. Some add­ons let the tractor make very precise turns without the driver even touching the steering wheel. Others uses radio base­stations set up around fields to provide even more accurate navigation than satellites.

And John Deere isn't the only company selling this kind of tech; its main competitor, Case IH, markets similar systems as do a slew of lesser­known companies. Some are going even further than modifying existing tractor designs: The aptly named Autonomous Tractor Corp. designed a fully autonomous tractor prototype that looks like a golden boxy tank, but without a seat for a driver.

The systems are pricey: Outfitting a new tractor with top­of­the­line auto­steering, navigation and guidance tech could cost upwards of $20,000, Reed said. There are also activation and subscriptions fees if farmers want to use the company's satellite or radio signals.

And there are risks to reliance on software. Poole said that one neighbor using self­driving technology downloaded a software update that disabled his tractor for a week in the middle of planting season this spring. If a system is working, farmers will often hold off on updates rather than risk complications, he said.

Those gaps could raise added cybersecurity risks. John Deere takes this issue seriously, Reed said, and encrypts its systems to protect them from hackers.

Still, issues like digital security fears, along with more traditional physical safety concerns, make it hard for consumer automakers to get their self­driving vehicles past various regulators and on to public roads. But tractor makers have shown that much of the technology needed to fulfill the promise of autonomous vehicles is already here.

"All of the things we're doing on the farm will find their way into the consumer market in the coming years," Reed said.

Andrea Peterson covers technology policy for The Washington Post, with an emphasis on cybersecurity, consumer privacy, transparency, surveillance and open government.

Drought Is the Mother of Invention on the Farm

Source: http://www.wsj.com/articles/drought-...arm-1437041211

July 16, 2015 6:06 a.m. ET

Growers are quicker to invest in modern water-saving technologies, once a hard sell

Dan Galbraith shows the top an underground water sensor in a Limoneira lemon grove near Terra Bella, Calif.
PHOTO: DAVID MCNEW FOR THE WALL STREET JOURNAL

DroughtisMotherofInventionFigure1.png

It has been two decades since Willie Hartman started making farm equipment that helps growers conserve water, but sales didn’t take off until the past four years. Credit goes, in part, to California’s worst drought in at least half a century.

Farmers in the Golden State’s $45 billion agricultural economy are increasingly thirsty for ways to save water, prompting a burst in demand for tools and processes like Mr. Hartman’s On Target Sprayer.

At a cost of $20,000 to $50,000 each, Mr. Hartman’s machines electrically charge mists of pesticides and plant nutrients in a way that causes them to stick to leaves more evenly, using about 80% less water than conventional chemical-spray rigs. Sales at his Progressive Grower Technologies Inc. are rising 30% a year, more than triple the rate four years ago, he said.

“Everybody’s aware of water now,” Mr. Hartman said. “Everybody’s watching it.”

Farmers in the arid West long have sought to lower their costs, including for water. But it often has taken a back seat to pressing concerns about labor, energy and other expenses. Now a drought that has stretched for four years is heightening attention to water efficiency by California growers of crops, whether almonds, olives or peaches—and opening a new market opportunity for those peddling novel solutions and others whose products received little notice until now.

New technologies range from a nontoxic gel made by Moasis Inc. that, mixed into soil, stores and slowly releases water near roots, to companies using satellite and aerial imagery to help farmers better plan how to irrigate crops based on how sections of their fields are faring. A water-pump monitor made by PowWow Energy Inc. alerts farmers to leaks in irrigation equipment.

Their emergence underscores the resilience of a U.S. agricultural sector long accustomed to the vagaries of weather but under increasing pressure in recent years from foreign growers that sell to American supermarkets as well as rising costs and regulatory mandates. Along with leaving fields unplanted and diverting water to higher-value crops like berries or nuts, farmers in California—the nation’s biggest agricultural producer and exporter—are implementing robotics and other techniques for boosting productivity.

It isn’t clear whether the nascent water technologies can significantly dent water use and soothe concerns about availability in the near term. But some farmers and water-efficiency advocates say adopting new processes and technologies could cut agricultural water use by 10% to 25%.

A preliminary study in June by the Center for Watershed Sciences at the University of California, Davis, estimated the drought would cause $2.7 billion in statewide economic losses this year from farmers fallowing more than a half-million acres of land and 18,600 agriculture-related job losses. California recently ordered holders of some of the oldest and most significant water-rights claims in the state, many of whom are farmers, to stop drawing water.

Thistles grow in an abandoned vineyard near Terra Bella, Calif., where drought is in its fourth year. PHOTO:
DAVID MCNEW FOR THE WALL STREET JOURNAL

DroughtisMotherofInventionFigure2.png

Daniel Sumner, an agricultural economist at UC Davis, said that people focused simply on cutting absolute water use are looking at it the wrong way. Growers who find new ways to produce more food per unit of water may generate bigger gains in water efficiency, he said. “If we can grow more food with our water, that’s presumably what we would like to do,” he said.

One of the biggest ways that is happening is through the reams of agricultural data seeping into water technology.

Growers say they are regularly receiving sales pitches for new tools that measure everything from soil temperature to air pressure to help better understand how much water their crops need.

In late 2013, Terlato Wine Group, a large producer of California wines, began testing one new system from Tule Technologies Inc. on 130 of its 600 wine-grape acres, said Vice Chairman John Terlato. Through its wire sensors, Tule quantifies how much water the plants actually use day to day by measuring the water they release as vapor. It has allowed Terlato to irrigate its vines with the amount they need when they need it, yielding higher-quality grapes and reducing water usage by 20% on those acres, he said. This year, Terlato is nearly tripling its Tule acreage.

Mr. Terlato’s company has scrutinized its water use for years. Yet Mr. Terlato, like many growers, is irked by a generalized perception in California that farmers are taking more than their share of water.

The “big, wide brush that everybody is using too much water is a generalization, and I know there are a lot of people working hard out there to be thoughtful about this,” he said.

San Francisco-based Tule, founded in December 2013, has sold more than 320 systems so far, many to wine vineyards. Its sensors, which each cover about 10 acres of land, cost $1,500 apiece.

For some growers, water-conservation investments also are a strategic calculation. Publicly traded Limoneira Co. , with about 7,300 acres of citrus and avocados, is preparing now for long-term water limits it expects will eventually be based on historical usage rates.

For about seven years, it has carpeted its groves with mulch made from yard waste and construction debris to stem soil-moisture evaporation, cutting water use by between 15% and 25%, said Chief Operating Officer Alex Teague. But last year, the Santa Paula, Calif., company installed plastic tubes that collect water samples as far as two feet below the ground on 400 of its acres in a pilot project to better monitor nutrient and moisture levels. Limoneira also this spring began testing a new irrigation system by Netafim Irrigation Inc. Instead of the fan jets Limoneira currently uses, which sit 12 to 24 inches from the ground and lose about 15% of their water to evaporation, it is a system of tubes, dotted with holes, that lies on the ground. It can pump a higher volume of water during the hottest parts of the day directly to trees and lose less water to evaporation because it is closer to the ground.

Mr. Teague estimates the combination of better monitoring and new irrigation could save an additional 25% of water use when deployed throughout the company, and come in handy soon. By lowering water needs now, “you’re giving yourself a cushion” for future dry spells, he said.

Write to Ilan Brat at ilan.brat@wsj.com

NEXT

Page statistics
2913 view(s) and 125 edit(s)
Social share
Share this page?

Tags

This page has no custom tags.
This page has no classifications.

Comments

You must to post a comment.

Attachments