Applying Valuable Lessons from the DIUx xView 2018 Detection Challenge

In my last post, I shared some of the lessons learned by the Wovenware Data Science team this summer as a contestant in the DIUx xView 2018 Detection Challenge. This contest asked participants to create innovative solutions for national security and disaster response issues using one of the largest publicly available datasets of overhead imagery, xView.

As mentioned previously, as part of the challenge we trained object detection models, packaged them inside Docker containers, and gave the containers a private validation cluster provided by DIUx.

Some of the first lessons we learned revolve around the fundamental aspects of machine learning preparation, including the need to wisely select tools and techniques needed for effective dataset exploration, cleansing and preparation; as well as the fact that working with dense, unbalanced datasets requires really strong time management and planning.

In addition, below are other key lessons we learned from the DIUx xView 2018 Detection Challenge.

Semi-Automated Model Training Requires Hard Work

Model training was based on a Keras implementation of the Single Shot Detector (SSD) and FAIR’s Retinanet. We developed custom scripts for automating model training using YAML config files to specify hyperparameters for our experiments. To achieve automation of training we made use of default and custom Keras callbacks to monitor loss plateaus, perform learning rate adjustments, and detect over fitting. For training we used Wovenware’s Octoputer server, which is powered by a GPU cluster of NVIDIA Titan Xps and CUDA 8.

With this strategy in place we were able to perform over 200 experiments in the first two weeks. Subsequent training sessions pushed that number to approximately 300 total experiments. This large volume of experiments in need of validation, and subsequently, large volumes of results in need of interpretation, grew to an almost unmanageable amount. Because of this, we brainstormed possibilities for modeling results interpretations and model selection with the goal to aid data scientists dealing with such a large volume of results just like we were.

In the end we learned that pushing the boundaries of automated deep learning model training is extremely difficult. Although we found ways to imitate basic decisions made by data scientists in the field (such as stopping training after detecting over fitting) we still relied on a long list of manually designed experiments that evolved as results from previous experiments were explored. The bottom line is that when going after AI pipeline automation don’t expect it to be a breeze.

Validate Early and Often

The main performance metric used in the xView competition was DIUx’s custom implementation of mean average precision (mAP). We performed local validation of our models using 10% of the dataset as a hold-out set, but only after DIUx published a hotfix for a bug that caused classes to score NaNs when no bounding boxes were assigned to them. We also frequently submitted containerized inference scripts to their validation cluster to get constant feedback from the private hold-out set. This feedback was very important for us to constantly gain insight about our models’ performances in the wild and where we stood regarding the single CPU / 8GB of RAM / 72hr constraints. That is why relying on local scoring code was not enough – if we did not validate with an appropriate hold-out set our results were just a bluff.

It Pays to Continuously Document Results

One of the most important lessons we learned is that consistent results documentation is essential, or else much of your work is in vain. Due to the large volume of results we generated, proper documentation could make or break our progress. That is why we recommend keeping a well-documented log of what you tried and how you tried it, what worked out and what didn’t, how to go back and find model files in the future, and how to replicate something you already tried.

It is extremely important to always avoid progress paralysis by lacking structured data for analysis.

Conclusions

As you might have noticed, all of our lessons learned revolved around the popular ML life cycle, from data exploration to hypothesis formulation, from model training and validation back to data exploration. It is clear to us data scientists at Wovenware that we are ready for the next challenge. We are ready to apply the lessons learned.

Bringing it On During DIUx xView 2018 Detection Challenge

Wovenware’s Data Science team has been working with state-of-the-art Convolutional Neural Networks (CNNs) for some time now as part of our deep learning and machine learning AI projects.  To share this experience with others in the industry, we recently participated in the DIUx xView 2018 Detection Challenge. The contest asked participants to bring their best solutions that push the boundaries of computer vision, to develop new solutions for national security and disaster response.  It revolved around the use of one of the largest publicly available datasets of overhead imagery, xView.

The xView dataset contains images from different regions of the world annotated using bounding boxes. It was chipped from images collected with DigitalGlobe’s WorldView-3. Comprised of 846 high resolution satellite images in TIF format and 600K+ bounding box annotations belonging to 60 categories, xView is considered to be a dense, imbalanced dataset because each category can have anywhere from dozens to hundreds of thousands of annotations.

As part of the challenge, which took place April to August 2018,  we at Wovenware trained object detection models, packaged them inside Docker containers, and gave the containers a private validation cluster provided by DIUx. We did this iteratively for the duration of the challenge.

While this challenge was just that – quite challenging – we learned some very important lessons from our efforts.  Below are a few lessons learned, as well as our strategy moving through the competition and challenges we overcame along the way.

Tools and Techniques for Dataset Exploration, Cleanup, and Preparation Need to be Methodically Planned

Our first decision was what strategy to use for the dataset exploration, cleanup and preparation. We decided to perform visual inspection of the data using QGIS, geopandas and Jupyter Notebook. Next, we chipped out a custom dataset with various resolutions ranging from 300px2 to 1024px2 using the chipper code published by DIUx as part of its data utilities repository. After the dataset was created we were eager to immediately start training our baseline models, yet we noticed some anomalies in the dataset.

First, we encountered problems related to the integrity and availability of the dataset. Even though the dataset passed our local checksum tests we were missing data and a clean download did not become available for more than a week. This forced us to stop training our first round of models until we could get our hands on the complete dataset.

Before creating new datasets, we also decided to enhance the chipper code provided by DIUx to include the tail of each TIF, which otherwise was being ignored, resulting in chipped datasets that were missing features present in their original TIFs. Figure 1 below shows an example of a set of 512px2 chips produced by the vanilla DIUx chipper. As you can see, the image shows 81 cells in a 9×9 grid where each cell would become a chip and a tail which is not large enough to accommodate another row or column of chips. For dataset creation we added a sliding window over the tail that overlapped with chips from the last row and column. For performing inferences, we padded TIFs to make their size a multiple of the chip size. This latter approach was used because we were short on time to implement Non-Maximal Suppression on overlapping chips at inference time.

DIUx chipper output

Figure 1: Visualization of DIUx chipper output missing tail features.

We then decided to perform in depth manual and code-based explorations of the dataset to find other anomalies before resuming model training. This was done with custom scripts for removing corrupted annotations such as negative coordinates or coordinates that exceeded TIF dimensions, which would place the object outside the image. From this work, we found a lot of faulty bounding boxes, as did other xView participants. Unfortunately, mis-categorized training samples were impossible to detect without a full-on QA session of all the 600K+ annotations. After cleaning up our uncorrupted dataset we were finally ready to resume training.

Our eagerness for CNN training to begin got the best of us and rather than relying on checksums as the only way to verify data integrity, we began training before performing the full exploration, which proved to be a mistake. We also learned that it is hard to systematically detect all anomalous annotations in such a large dataset and we had to live with the fact that we could not detect nor remove incorrectly categorized bounding boxes from our training data.

The first lesson learned is aligned with an old saying – The Early Bird Catches the Worm – although in our case the worm is not a good reward.  Yet, early data exploration will most certainly reward you with early bug discoveries.

Working with a Dense, Unbalanced Dataset Requires Time Management and Planning

As I mentioned before xView is a dense, unbalanced dataset and our proposed solution heavily considered those factors. At first, we considered using either model ensembles; multi-headed object detectors for multiple subsets of the original 60 categories; and Focal Loss, which Facebook AI Research (FAIR) has used to improve state-of-the-art results of single shot detectors for dense, unbalanced datasets. In the end we only tried model ensembles and Focal Loss with Retinanet.

Our model ensembles were based on various lists of object classes grouped by either object type, object size, physical object appearance, or per class bounding box count. As you can imagine we ended up with a long list of potential experiments – in the hundreds – that only kept growing to ensure we always had more possible solutions to try. Experiment priorities were assigned based on estimated training times and the incremental nature of most of our experiments. The number of models submitted as part of any given ensemble was limited by their total execution time. This was because the DIUx established execution time constraints for all containerized submissions required that 282 TIFs be processed on a single CPU with 8GB of RAM in under 72 hours.

We learned that planning a wide range of experiments ahead of time helped prioritize our efforts in accordance with checkpoint deadlines and execution time constraints. Time management and planning was imperative because as Benjamin Franklin once said: “You may delay, but time will not.”

This is the first in a two-part series that explores the  valuable lessons learned as we competed in the DIUx xView Detection Challenge.  Leslie collaborated in the writing of these posts. Stay tuned for the next post, which explores how we achieved semi-automated model training, validation and results documentation.

Reaching Beyond the Stars to Glean AI-Driven Insights

When Yogi Berra said, “You can observe a lot just by watching,” he was on to something. It’s certainly the case when it comes to satellite images – where lots of information can be gleaned through observation from above – although that was probably the furthest thing from Yogi’s mind when he made that statement many years ago.

Through hundreds of eyes in the sky over the Earth, we are capturing and analyzing images that can be used to provide insights to industries as diverse as farming, oil & gas, urban planning and more. Early adopters in industries such as financial services are starting to reap the benefits, but the advantages are there for the taking for all types of businesses and organizations.

While satellites have been around since the Soviets launched Sputnik in 1957, for the last twenty or so years they have been used primarily by government and communications companies. In addition to military uses, the government has relied on satellites for scientific endeavors – from measuring radiation in the Earth’s magnetic field and observing weather patterns to exploring planets. Communications companies started using satellites for television broadcasting following the first satellite broadcast of the Tokyo Olympics in 1964.

If Satellite Imagery is the Eyes, then AI is the Brains

But what’s bringing about a new burst of interest in satellite imagery is its pairing with Artificial Intelligence (AI). The massive number of images that come from satellites aren’t worth anything if you can’t make sense of them. While satellite imagery gives us an elevated look at massive amounts of images, new AI-based algorithms are giving us the means to comprehend them — to mine, process and find patterns in the images.

Bringing Insight to the Images

But, there’s a lot that goes into turning images into insights and it requires an army of people to do it. You need to have data engineers to gather, store and put the data in a format that is useable for analysis, as well as make it accessible to data scientists, who in turn build algorithms to analyze and make sense of it. But before the data scientists can do that, they need to teach the intelligent application to recognize objects in the images, and distinguish for example, a car from a bus. That requires a great deal of time and effort, with data specialists, or private crowds, annotating and labeling huge volumes of images in order for data scientist to be able to use them.

The other consideration when it comes to using AI to detect images is that it’s never “one and done.” AI applications need to be continuously improved, or “taught” to be able to deliver the insights that companies are looking for.

The information you gain from these images can be used to quickly and accurately derive insight that would be nearly impossible to detect and analyze with the human eye and brain. It’s becoming a new way for companies to predict future outcomes and read between the lines. For example, it provides new sources of intel that could help hedge fund managers predict where markets are going and inform investment decisions. Satellite imagery can help them assess, for example, the health of the oil & gas industry by seeing how many cargo ships may be in a given port; or the shadow of oil tank lids, indicating the volume of oil that they contain. It could also be used to predict retail revenue by counting how many cars are in mall parking lots, among many other projections.

Benefiting from the technology today

Here are some other examples of how satellite imagery and AI are providing critical insights today:

  • Increasing crop yields – Microsoft is working with the International Crop Research Institute for the Semi-Arid Tropics (ICRISAT) to help farmers in India achieve better crop yields. Using a satellite that provides images on the stages of crop growth, along with weather information and AI technology, they created an AI-Sowing app to predict the best day to sow the crops to get a better outcome.
  • Protecting human rights – In addition to commercial uses, it is being used by nonprofits, such as human rights organizations, to expose abuses and conflicts, like detecting prison camps, mass graves and villages that have been burned.
  • Tracking economic growth – By monitoring the numbers of cars, electric lights in the night sky and construction, we can track the development and economic growth of countries around the world.
  • Locating solar panels – The solar industry is using satellite imaging to see where solar panels are located and based on that, along with weather data, they can determine which homes would be well suited for them. The city of San Jose, California is using this technique on a larger scale to locate where they could generate additional energy from new panels.

These major applications are currently underway, and many more innovative opportunities are on the horizon. When combined with AI, satellite imagery sheds light on key, previously hidden information that can lead to greater clarity and business growth. By leveraging such out-of-this-world insight, companies can bring their business planning and execution to a whole new level.

A Year After Maria: Lessons of Survival

September 20 marks one year since Hurricane Maria made landfall in Puerto Rico, just 14 days after the island was hit by Hurricane Irma. Maria, a category 5 hurricane, not only destroyed Puerto Rico’s energy grid and communications infrastructure, but it also upended many lives and left a permanent scar on everyone who survived it.

We are still living with its effects, from an unstable power grid that needs to be redesigned from the ground up, to a collective sense of PTSD that compels us to buy excessive amounts of water and fuel whenever a new storm forms in the Atlantic Ocean. But Maria also left us with an advanced degree in resilience. We learned how to quickly adjust to difficult situations and plow through. The most important lesson that came out of our experience: Don’t give up. Keep moving forward. It’s too easy to fall prey to inaction after a big disaster like a hurricane, but to survive we must resist that urge and act. Otherwise, things just get worse, and inaction leads to further inaction.

Here are five lessons I learned over the past year that stood out, especially in my role as an entrepreneur running a business:

1. Focus on people

First and foremost, make sure your employees and their families are safe and help them in any way you can. The day after the storm, we checked in on all of our employees to make sure that they and their families were safe. We continued to check in with them throughout the emergency and provided as much help as we could. For example, when drinking water was hard to get, we arranged for water gallons to be delivered to the office for everyone, and when cash was scarce, we distributed payroll cash advances. While it goes without saying that employees are the number one asset of any company, when a disaster strikes, the typical employer/employee relationship transforms and becomes one of humanitarian concern.

2. It’s all about communication

Make sure to communicate with your customers, partners and employees, early and often. There is a lot of misinformation, especially after a big disaster like a hurricane. It’s important that you accurately explain the situation to your customers, partners, and employees as well as the plans and courses of action you are putting in place. By letting them know what they can expect, you can help assuage any concerns they may have. Last year, it was heart-warming to see how our customers not only understood the gravity of the situation, but also opened their doors to us with office space and other offers of help.

3. Contingency plans are required

Never underestimate the need for a contingency plan and the importance of communicating it to all your stakeholders. While the contingency plan will likely evolve once a disaster strikes, much of it will still be applicable and it will help you to get things underway quickly.

4. You learn who your true partners are

In addition to communicating with customers, it’s important to share your situation with partners, yet not all of them may understand the true magnitude of your plight. We were surprised to learn that some may focus on your commitments to them and hold you accountable regardless of the circumstances, while other are completely understanding and offer help. You’ll probably spend more time managing concerns from some partners than customers who see the effort and continued commitment that you are providing them. In the end, you will quickly figure out who your true partners are.

5. Always help others, no matter what

Whenever possible, help others. After an event like a category 5 hurricane everyone is affected, and only by helping each other can everyone make it through. That includes competitors, too. When the hurricane hit, we all shared resources and the Internet with partners and competitors alike. And, no amount of help is too small — even if it seems trivial to you, it may make a big difference to whomever you help.

 

Finally, after looking back at this past year and talking to entrepreneurs who were impacted by Hurricane Katrina in New Orleans, I have concluded that the major difference between those who make it in a disaster and those who don’t is mainly based on one thing: never giving up.

Using AI to Classify and Swat at Disease-Carrying Mosquitoes on Post-Hurricane Puerto Rico

Hurricane Maria caused some major damage to Puerto Rico. The lack of power, food and water, along with the damage to infrastructure, were well reported and experienced by us all. But among other lingering effects of the hurricane, was the explosion of disease-carrying mosquitoes caused by stagnant water, which is becoming much more than a pesky problem.

Because of this, the work that the Puerto Rico Science, Technology & Research Trust and its Vector Control Unit (PRVCU) has been doing since 2016 is more critical than ever. It has been working to help prevent and manage diseases spread by mosquitoes, to gain an understanding of why many mosquitoes have become immune to insecticides approved by the FDA. In order to do this, researchers have spread out across the island, capturing different mosquito species in traps; monitoring and testing them for viral presence and insecticide resistance; and labeling and classifying them.

To manually capture and classify thousands of mosquitos, across different areas in Puerto Rico is extremely time and knowledge intensive, as you can imagine. So, earlier this month we announced that we’ve begun training a machine learning algorithm, which will automate the time-consuming task. Through our private crowd of data specialists, we’re identifying and labeling thousands of images of mosquitoes and data sets, confident that this AI automation will save the PRVCU months of manual work.

While we often hear about the use of AI in the business world – within insurance firms, banks, e-commerce and manufacturing, this mosquito project just goes to prove how far-reaching, innovative and endless are the possibilities of AI beyond the office walls or factory floor. Once you have the AI bug, you see the many ways it can be used for good – one algorithm at a time.

Hitting The Need for Diversity Head On in Forbes Post

In a recent Forbes blog post, I posed the question: Is there such a thing as a prejudiced AI algorithm? It’s a topic that’s been getting lots of attention by the media, as well as tech leaders lately, as complex AI solutions are taking on a life of their own and making decisions for us.

But what continues to make AI tick is the data that is fed to them, and that data is often delivered by real-life people who may have their own preconceived opinions, experiences and maybe even their own biases. As I outline in the blog post, the only way to ensure that an algorithm is trained to make fair and unbiased decisions is by making sure there is broad perspective and experiences among the people that will train it. Diversity is key to the data science team.

As I mention in the post, there are a few ways to make sure that the data science teams brings diversity to the training, including carefully reviewing candidate qualifications that go beyond the tech credentials; considering outsourcing parts of the project to get perspective beyond your company; offering internships and other programs to engage the next generation of professionals; and investing in training programs.

This Forbes post sheds light on a topic that is of great concern to us here at Wovenware and we’ll continue to explore ways to ensure any AI solution we create supports fair, unbiased and ethical decision making. The future depends on greater awareness and acceptance of diversity in the workplace and the creation of AI is no exception.

We’d love to hear your thoughts. Please reach out to us: info@wovenware.com