The Round-Robin Exercise of the Fire CCI Project has now officially begun. The Round-Robin exercise of the fire_cci project has officially kicked off on 1st November 2011 and will last three months. The objective of the Round-Robin (R-R) Exercise is to compare the burned area (BA) products using different external algorithms and identifying the one showing the best performance (internal and external). To evaluate the performance of as many BA algorithms as possible the exercise will be oriented to facilitate and encourage the participation of as many external researchers and institutions as possible.
The Round-Robin exercise was introduced to the International Fire Research community on 17 – 18 October 2011 in Stresa, Italy. Thanks to the discussion and suggestions made by experts the R-R exercise has been improved to now meet the demands of the research community.
According to the suggestions made at the technical workshop two modes of participation were defined:
The fire_cci project has defined a certain database available to the R-R participants. On the other hand the developers will apply their algorithms with the available datasets to produce burned area. The BA maps will be validated using the reference data and the selected metrics. Based on the validation assessment results the algorithm with the highest performance will be established and the analysis will be published in the respective report.
The R-R dataset focuses on four study sites located in Canada, Brazil, Kazakhstan and Australia as well as two years, 2005 and 2006 respectively. Applicable sensors for this project are AATSR, VGT and MERIS. The study sites were chosen for their contrasting ecosystems and fire regime conditions in order to demonstrate the spatial consistency of the BA product. The years 2005 and 2006 were chosen as they show contrasting fire occurrences within the study areas, thereby enabling the participants to test the algorithms inside as well as outside fire affected regions.
Full participants are granted access to reference data regarding the four test sites consisting of the 2005 and 2006 fire perimeters. The BA algorithms will be calibrated using the 2005 reference data. The 2006 data will then be used for the purpose of validation assessment in order to assure independence between the final product and the reference data used.
The Round-Robin participation process
The Round-Robin Exercise is a public and open process which benefits from the knowledge of the research community who works in burned area detection and mapping. Consequently, the fire_cci project welcomes every researcher and institution with the will to participate and contribute with their BA algorithms to the R-R exercise.
First, fire_cci considers the option of registering as a “exploratory participant” who will be allowed to download the pre-processed data of the three sensors (AATSR, VGT and MERIS) for the four study sites (Canada, Brazil, Kazakhstan and Australia) and two years (2005, 2006) selected for the R-R exercise. The participants can register in the webtool at http://www.fireccidataproducts.org/.
If after the evaluation of the pre-processed data the “exploratory participant” wants to use them to produce BA and upload the results to fully participate in the R-R, he will have to accept the Data Policy first. When the Data Policy is accepted the “fully participant will have access to the reference data. Since the fire perimeters of 2006 will be kept for the validation assessment, the participants will be able to upload as many BA maps as they want to check the performance of their algorithms in the 2006 data. From the validation assessment results an algorithm ranking will be established and the best performing BA algorithm will be selected for the global BA product.
Timelines and deliverables
The R-R exercise was announced during the Global Burned Area Algorithms Workshop on the 17th-18th of October 2011. However, the official kick-off of the R-R exercise will be on 1st November. The timeframe of the R-R exercise will last three months, so the timeline to upload all deliverables (BA products and technical report) will be 31st January 2012.
During the time the R-R is open the participants have free access to the data available to the R-R exercise and can use the tools of the on-line platform.
At the end of February 2012 the Product Validation Team will deliver the Product Validation and Algorithm Selection Report (PVASR). The document will be publicly available and will be sent to all the participants. A detailed description of the R-R database, instructions for participation, including user guidelines for the on-line platform is documented in the Round-Robin Database Document.