Snapshot Wisconsin relies on different sources to help classify our growing dataset of more than 27 million photos, including our trail camera hosts, Zooniverse volunteers and experts at Wisconsin DNR. With all these different sources, we need ways to assess the quality and accuracy of the data before it’s put into the hands of decision makers.
A recent publication in Ecological Applications by Clare et. al (2019) looked at the issue of maintaining quality in “big data” by examining Snapshot Wisconsin images. The information from the study was used to develop a model that will help us predict which photos are most likely to contain classification errors. Because Snapshot-specific data were used in this study, we can now use these findings to decide which data to accept as final and which images would be best to go through expert review.
Perhaps most importantly, this framework allows us to be transparent with data users by providing specific metrics on the accuracy of our dataset. These confidence measures can be considered when using the data as input for models, when choosing research questions, and when interpreting the data for use in management decision making.
The study examined nearly 20,000 images classified on the crowdsourcing platform, Zooniverse. Classifications for each specie were analyzed to identify the false-negative error probability (the likelihood that a species is indicated as not present when it is) and the false-positive error probability (the likelihood that a species is indicated as present when it is not).
The authors found that classifications were 93% correct overall, but the rate of accuracy varied widely by species. This has major implications for wildlife management, where data are analyzed and decisions are made on a species-by-species basis. The graphs below show how variable the false-positive and false-negative probabilities were for each species, with the whiskers representing 95% confidence intervals.
Errors by species
We can conclude from these graphs that each species has a different set of considerations regarding these two errors. For example, deer and turkeys both have low false-negative and false-positive error rates, meaning that classifiers are good at correctly identifying these species and few are missed. Elk photos do not exhibit the same trends.
When a classifier identifies an elk in a photo, it is almost always an elk, but there are a fair number of photos of elk that are classified as some other species. For blank photos, the errors go in the opposite direction: if a photo is classified as blank, there is a ~25% probability that there is an animal in the photo, but there are very few blank photos that are incorrectly classified as having an animal in them.
Assessing species classifications with these two types of errors in mind helps us understand what we need to consider when determining final classifications of the data and its use for wildlife decision support.
When tested, the model was successful in identifying 97% of misclassified images. Factors considered in the development of the model included: differences in camera placement between sites; the way in which Zooniverse users interacted with the images; and more.
In general, the higher the proportion of users that agreed on the identity of the animal in the image, the greater the likelihood it was correct. Even seasonality was useful in evaluating accuracy for some species – snowshoe hares were found to be easily confused with cottontail rabbits in the summertime, when they both sport brown pelage.
Not only does the information derived from this study have major implications for Snapshot Wisconsin, the framework for determining and remediating data quality presented in this article can benefit a broad range of big-data projects.
The Snapshot Wisconsin team is often asked why we accept data only from our Snapshot-specific cameras. While there are several reasons, the reason that was highlighted in the April 2019 newsletter was because Snapshot Wisconsin cameras are programmed to take a single photo at 10:40 a.m. each day. Although 10:40 may seem like an arbitrary time, this corresponds to the approximate time that a NASA satellite flies over Wisconsin and collects aerial imagery. (More information on how NASA data and Snapshot data are complementary can be found in this blog post.)
It may be difficult to recognize the value of a blank photo in wildlife research, but a year-long series of these photos allows us to examine something very important to wildlife: habitat condition. For each camera site, the time-lapse photos are loaded into the statistical software, “R,” where each pixel in the image is analyzed and an overall measure of greenness is summarized for the entire photo. That measure, called the Green Chromatic Coordinate, can be used to identify different “phenophases,” or significant stages in the yearly cycle of a location’s plants and animals. These stages can be delineated on a graph, called a phenoplot, where a fitted curve reveals the transition day-by-day. The 2018 phenoplot for one Snapshot Wisconsin camera site is seen below.
In 2018, 45 camera sites had a complete set of 365 time-lapse photos, but we expect many more sites to be included in the 2019 analyses. The relatively small sample size for 2018 is due in part to many counties not being opened for applications until partway through the year, but also because time-lapse data are rendered unusable if the date and time are not set properly on the camera. This may happen when the operator accidentally sets the time on the 12-hour clock instead of the 24-hour clock, or if the hardware malfunctions and resets the date and time to manufacturer settings—this is why we ask our volunteers to verify the camera’s date and time settings before leaving the site each time they perform a camera check.
The information derived from these analyses will be integrated into wildlife models. For example, the objective of one ongoing DNR research project is to understand linkages between deer body condition and habitat, which includes what’s available to deer as forest cover and food resources, as well as weather-related factors, such as winter severity or timing of spring greenup. The project currently uses weather data collected across the state to estimate snow depth, temperature, and winter severity, and creates maps based off this information.
Snapshot’s time-lapse cameras offer a wealth of seasonal information regarding type of forest cover and food sources, as well as weather-related information. In the future, phenological data obtained from Snapshot cameras could be used to create “greenup maps” that provide estimates of where and when greenup is occurring, and potentially test that information as a means of better understanding how environmental factors affect deer health, such as whether an early spring greenup improved deer body condition the next fall.
One of Snapshot Wisconsin’s major goals is to alleviate some of the burden associated with time-consuming in-person survey techniques. This is possible because trail cameras can serve as round-the-clock observers in all weather conditions. Annual Greater Prairie-Chicken lekking (breeding) surveys were identified as having good potential to be supplemented by Snapshot Wisconsin cameras, and a pilot study was conducted in spring 2018.
The Greater Prairie-Chicken (GPC) is a large grouse species native to grassland regions of central Wisconsin. During the breeding season each spring, males compete for female attention by creating a booming noise and displaying their specialized feathers and air sacks. This ritual occurs on patches of land known as leks, as seen in the photo above. Wisconsin DNR Wildlife Management staff identify leks in the early spring and return to each site twice in the season to count the number of booming males. The number of males present on the leks is used as an index to population size. Three Snapshot Wisconsin cameras were deployed on each of five leks – one camera facing each direction except for east to reduce the number of photos triggered by the rising sun. The cameras were deployed from late March through mid-May, and all in-person surveying was conducted within the same period.
As seen in the graph above, Snapshot Wisconsin trail cameras recorded male GPC at all five of the study sites. This is significant because GPC were only detected on three of the five leks according to the in-person surveys. On leks A, B, and D, where both in-person and camera surveying detected GPC, the in-person maximum of male GPC was higher. However, when the trail camera maximum is averaged across all survey days, the maximum is nearly the same for both survey methods (8.5 in-person, 8.3 trail camera).
In-person surveying requires the observers to arrive before dawn and remain in the blind until after the early morning booming has finished. Snapshot Wisconsin cameras record the hourly activity on the lek while minimizing the risk of disturbance due to human presence. The graph above displays the total number of male GPC photos captured by hour and shows a small uptick in photos around 7 p.m. Because the in-person surveys do not include evening observations, Snapshot Wisconsin data offer a way to examine the lek activity at all hours.
Additionally, continuous data collection is not only useful in capturing the activity of GPC, but offers insight into the dynamics of Wisconsin’s grassland ecosystems. In total, Snapshot Wisconsin cameras collected over 3,000 animal images including badger, coyote, deer, other bird species, and more. Some photos were even a little surprising. Pictured above is a coyote just feet away from prairie chicken. We might expect the GPC to flee in the presence of a predator, but this one appears to be standing its ground. In the upcoming pilot year two, we hope to gather even more information about the interactions within and among species found on these leks.
One of the major Wildlife Management implications for Snapshot Wisconsin is the project’s contributions toward a system the DNR uses to calculate the size of the white-tail deer population in Wisconsin. Fawn-to-doe ratios, or FDRs, are found by dividing the number of does by the number of fawns seen during the summer months and are summarized by the (82) management units across the state.
In total, three programs contribute to FDR estimates: Snapshot Wisconsin, Operation Deer Watch, and the Summer Deer Observation Survey. An advantage of incorporating Snapshot Wisconsin data in these estimates is that Snapshot cameras tend to be placed in secluded, natural areas, whereas the other two collection methods are opportunistic, meaning they’re biased toward counting deer seen near roadways.
One challenge associated with trail camera data is that the same individual animals may walk by the camera multiple times throughout the data collection period. To account for this, we average the total number of does seen in photos with at least one doe, and then average the total number of fawns in each photo containing at least one fawn. We then take the average number of fawns and divide it by the average number of does.
Fawns and does may or may not be in the same photo to contribute to their respective averages. Defining a single camera-level average for each site drastically reduces the amount of data involved but ensures that the FDR is not skewed toward does, which tend to appear much more frequently on Snapshot cameras.
The above maps show the camera sites that contributed to FDR estimates in 2017 and in 2018. Photos from exclusively July and August were analyzed. A site only contributes to the estimate if there were at least 10 doe observations in one of the two months, but can be counted twice if it had at least 10 doe observations in both months. Statewide, 897 cameras contributed to 2018 FDR estimates, a 44% increase from the 622 sites that contributed in 2017. Some deer management units decreased in sample size from 2017, but
Above are the results of the 2017 and 2018 FDR estimates using Snapshot Wisconsin data. Only deer management units with a minimum of 5 camera sites were included in the analysis. In 2018, the range of FDR was 0.75 – 1.2, which is an overall increase from the range of 0.62 – 1.13 in 2017. Snapshot Wisconsin was launched statewide in August 2018, meaning most cameras in the newly open counties were not deployed until after the data collection period. We expect that the number of cameras in the 2019 analysis will increase again, which would give us even more accurate estimates.
For January’s Science Update, also featured in The Snapshot monthly e-newsletter, we explored the accumulation of Snapshot Wisconsin photos over time and how the number of photos taken fluctuates with the seasons. To date, our data set contains more than 24 million photos, and their content is a vital component of the Snapshot Wisconsin project.
The bar chart above indicates that over half of the photos are blank. This can be attributed to the fact that our cameras contain a motion trigger function, which is designed to capture wildlife as it moves through the frame. However, this mechanism only detects movement and cannot differentiate between animals and vegetation. This means that on windy days during the spring green up period, thousands of blank photos can be captured. Occasionally cameras will malfunction and continuously take blank photos without being triggered by motion. This issue was more prevalent with earlier versions of our cameras; the model we currently use does not take as many blank photos. Additionally, over time volunteers have learned that trimming vegetation in front of their camera helps prevent blank photos.
Every day at 10:40AM, the cameras are programmed to record a time lapse photo. This is not only to document the “spring green up” period and the “fall brown down” period, but also to sync ground-level measures of greenness with satellite data. These photos are primarily used by our partners at UW-Madison and compose 7% of our data set.
It is not uncommon for our trail camera hosts to trigger the camera themselves during check events, which is the cause of most of the 3% of photos that are tagged as human. Although these photos are removed from the data set prior to analysis, they can be helpful in instances where the camera has been recording photos with the wrong date and time. A photo of a hand in front of the camera combined with the date and time reported by the volunteer at each check event are enough for us to adjust the date and time for the whole set of photos.
Twenty percent of the Snapshot Wisconsin photos are untagged, meaning they have yet to be classified as blank, human or animal. Many of these photos will be sent to the crowd sourcing website, Zooniverse, for classification. We hope to implement a program to automatically classify photos to work through this backlog as well.
Finally, about 14% of Snapshot Wisconsin photos are of confirmed animals. In the graph above, we have broken down which species appear in these photos. Deer are by far the most common species, appearing in about two-thirds of photos, followed by squirrels, raccoons, turkey, cottontail rabbits, coyotes, and elk. The remaining 8 percent of animal tags are divided up across 34 categories including other bird, opossum, snowshoe hare, bear, crane, and fox. Elk may have a higher proportion of triggers than expected because Snapshot Wisconsin cameras are placed more densely in the elk reintroduction areas than in other areas of the state.
We’ve gotten some great questions from volunteers on species distributions. One from early in the project was, “Do the ranges of gray fox and red fox overlap?” We couldn’t answer that at the time since there is no comprehensive tracking effort for gray fox in Wisconsin. Great news: we now have enough data from Snapshot Wisconsin photos that we can start shedding light on questions like this!
So far, we’ve had 6099 photo subjects classified as canids on Zooniverse from photos taken at 484 cameras. Of these, 5832 classifications from 465 cameras had enough agreement among users that we feel confident in these classifications, while 267 classifications from 19 cameras need review by experts before a final classification is determined.
Do we find different species of canid at the same camera site? Yes we do, but some combinations are more commonly found than others. The below graph shows that coyotes are the most commonly seen canid in Snapshot Wisconsin photos, and most cameras capturing canids have so far only captured coyotes. The most commonly seen multi-species mixes are coyote and fox. We’ve captured relatively few photos of wolves so far, but most cameras that have captured photos of wolves have also captured coyotes and/or fox. (Note that cameras in the elk areas are not included in this graph, since those cameras are more clustered than our other cameras and are not representative of the state.) Click on the graph to view a larger version.
The below map shows the canid data summarized by county. Data from the elk areas are included here and seen in the three small, square polygons. Note that since we do not have cameras in all parts of the state, and since different cameras have been active for different amounts of time, a lack of sightings in an area does not mean that a species is absent there – just that we haven’t seen it on our cameras (yet)! For example, we know from other data sources that wolves occur in more northern counties than what we’ve found on Snapshot Wisconsin cameras so far.
What we can say about these data so far:
- Coyote, gray fox, and red fox are found across the state.
- Photos of gray fox and red fox are sometimes captured on the same camera, and their ranges appear to have considerable overlap.
- Wolves are very infrequently detected compared to the other canid species.
As always, as we continue to expand the Snapshot Wisconsin program, we’ll be able to fill in more of the spaces in the map!
Our July #SuperSnap was all about fishers, and we’re just going to keep on rolling on the fisher train! This science update was inspired by recent comments on a photo of a fisher in central Wisconsin. The location of the photo might cause confusion if you base where fishers *should* be on the range map we have posted. The map shows fisher range extending to only the very northern part of the state:
Whereas we’ve seen fishers on Snapshot Wisconsin cameras in counties pretty far south:
In the case of a species like fisher, which was reintroduced to Wisconsin in the 1950s and expanded its range quickly, static distribution maps go out of date quickly. This brings up a larger point about range maps being inaccurate because they are based on old, incomplete or faulty data. We provide range maps to give volunteers an indication of where they are more likely to find a certain species, but these maps are by no means perfect. The fact that we do not have very good statewide data on the distribution of most species is indeed a major reason for starting a project like Snapshot Wisconsin!
Note that the above map shows counties where we’ve seen Snapshot Wisconsin photos correctly classified as fisher. Many of the gray counties do not have any Snapshot Wisconsin cameras and so we do not have any photos there yet. This is not to say there are no fishers in the gray counties!
Thanks to a dedicated effort by our volunteers, Wisconsin DNR staff and University of Wisconsin students, we were able to classify all of the elk photos from 2016!
This Science Update features data from the Clam Lake elk area only, due to a lack of elk photos from the Black River Falls area. From GPS collar information, we know that many of the Black River Falls elk prefer to hang out outside of our camera area (perhaps they are bashful?). When we have more Snapshot Wisconsin cameras in the counties surrounding Black River Falls, we hope to have enough data for a Science Update on those elk as well.
There were 120 cameras active in the Clam Lake area in 2016, capturing 3,996 triggers containing elk. After grouping consecutive triggers showing the same elk, we ended up with 305 unique elk events.
We graphed daily activity patterns of antlerless elk and bulls from the 305 unique elk events. Overall, elk were most active between 6 and 9 AM and 5 and 6 PM. Antlerless elk were most active around dawn and dusk, while bull activity peaked later in the morning and evening.
We also graphed monthly elk activity throughout 2016. Because not all of our cameras were active during the entire year, we corrected photo hit rate based on the percentage of cameras active each week. The image below shows this corrected photo count for antlerless elk and bull elk throughout 2016.
The marked spike in bull activity at weeks 36 through 40 indicates the annual rut period. That period corresponds to a sharp drop off in activity level for antlerless elk; cows tend to stay put during that period while bulls move around more. (Curious about why this might be? Click here for more information on elk life history and mating behavior.) The trail cameras give us the ability to pinpoint the time frame of the rut period more precisely than we were previously able.
Because Snapshot Wisconsin trail cameras put a time and date stamp on each photo, we are able to capture the diurnal (daytime), nocturnal (nighttime), and crepuscular (active early and late in the day) behavioral patterns of different species. The graphs below show daily activity patterns using the 24-hour clock for three categories of animals captured on Snapshot Wisconsin trail cameras in Iowa and Sawyer Counties from June 1 – September 7, 2016.
Bears were most active during the day and used the midday hours more than any of the other large mammals, while coyotes and deer showed the strongest crepuscular behaviors:
Porcupines were most active in the early morning hours before sunrise. Mustelids were uniquely active during a short portion of the early daytime hours:
Grouse activity was fairly steady through the day while turkey activity increased as the day progressed:
Food for thought: why might it be beneficial for animals to be more active during certain times of the day and not others?
Each year, the WDNR uses fawn and doe counts from August and September to calculate a fawn-to-doe ratio and estimate the size of the deer herd in Wisconsin. We get the fawn-to-doe ratio by dividing the number of fawns by the number of does. In 2015, the statewide fawn-to-doe ratio was 0.89, meaning there were about 9 fawns for every 10 does. Of course, this number varied a lot across Wisconsin.
Counts submitted by the public via Operation Deer Watch and by WDNR biologists are the primary data we use to calculate fawn-to-doe ratios. This information is very useful but somewhat biased, since observations are made during daylight hours and mainly along roadsides. Snapshot Wisconsin trail cameras give us a new way to count deer because the cameras operate all the time and are placed in more natural spaces.
In our first attempt to use Snapshot Wisconsin trail cameras to calculate fawn-to-doe ratios, we used Snapshot Wisconsin photos from August 2016 that were classified as deer by trail camera volunteers in Iowa and Sawyer Counties. There were 211 deer pictures from 13 cameras in Iowa County and 331 deer pictures from 13 cameras in Sawyer County. This is a very limited sample but it let us look for early patterns.
One thing was immediately obvious: we see the same does and fawns over and over again at each camera site. Before we could come up with any accurate estimates, we would have to account for repeated counts of the same deer. One method we tried was to use the maximum number of fawns and does seen in any single photo from each camera site. This leaves us with a much smaller number of deer observations at each site, but ensures that we do not over-count. When using this method, we end up with preliminary fawn-to-doe ratios between 0.7 and 1.0 that are close to what we would expect.
Stay tuned for more on fawn-to-doe ratios and other results as we continue to add photos and classifications!