Drone Data Processing Guide

This guide will explain how to use this code to process your drone data.

The first step is to acquire a large CSV file containing raw drone data. If you do not have one yet, follow the steps in Drone Data Extraction and Preprocessing first.

Directory structure

This code depends on the existence of several specific directories. If you cloned it from GIT, you probably already have them. The directory structure is as follows:

  • csv_files: contains input CSV files, direct from DatCon
  • csv_out: contains processed CSV files
  • plots: yep, contains plots
  • npz: for numpy zip files

Reading CSV files

The first step is to take the enormous CSV you made in Drone Data Extraction and Preprocessing and cut it to contain only the columns you need. Place all of the CSV files from a flight in the csv_files directory. Then, run load_csv.process_files() on a list of your filenames. Filenames are given with no directory and no extension. For example, passing in [‘FLY170’] will read ./csv_files/FLY170.csv

load_csv.process_files() will save a new CSV in the csv_out diretory. You can load in this data using pandas’ read_csv method:

pd.read_csv("./csv_out/'+NAME+'_processed.csv", sep=",", header=0)
df = df.assign(timestamp = pd.to_datetime(df['timestamp'].values))

The second line is necessary because read_csv reads datetimes as strings, and we need to convert them back to np.datetime64.

Processing data

At this point you should have a pandas DataFrame containing the following columns:
  • Lat
  • Lon
  • Hmsl
  • timestamp

columns provides a variety of functions for adding useful information to your DataFrame.

To take a moving average, which smooths the data a little, use columns.fill_avMov(). To convert from latitude and longitude to a local coordinate system, run columns.make_ENU() or columns.make_az_el(). You must provide an origin for this coordinate system, which I usually do visually (see the next section on Producing Plots). To label if the drone is moving or not at every point in time, use columns.fill_moving().

Producing Plots

Now that you have a DataFrame, you can plot it!

plotting.plot_latitude(), plotting.plot_longitude(), plotting.plot_height(), and plotting.plot_satellite() each plot the entire DataFrame, and accept figsize and color arguments to be passed to matplotlib.

If you want to plot a time interval, make a slice of the DataFrame using columns.chop_time(). This is especially useful to get rid of the GPS glitches at the beginning of most flights.

In order to read coordinates from a plot, add “%matplotlib notebook” before a plotting command. This will add a coordinate readout to the lower right corner of the plot. Use this for finding the timestamps for tagging points, and for finding the origin of your coordinate system.

There are also two more complex plots:

plotting.plot_moving() can only be used after running columns.fill_moving(). It plots points where the drone is moving in blue, and stationary points in magenta. This is used to tag points (see next section).

plotting.plot_points() can only be used after tagging points, and will plot each point in a different color, on a satellite map background.

Point tagging

points.mark_point() allows you to label a region of the DataFrame as a single physical point. points.mark_point() accepts a single timestamp as input. Make a plotting.plot_moving() plot, and find the timestamps at the approximate centers of stationary points. Then, points.mark_point() will begin from the timestamp you provide, and expand the radius to capture the whole time that the drone was stationary. DIAGRAM TODO?

points.stationary_stats() and points.point_stats() can provide statistical data about a time interval or numbered point.

Producing output files

To produce output files, use load_csv.output_df(). It will make a CSV containing all your data, and an NPZ with only a few columns.

To output point locations, use points.output_points(). It will make an NPZ with the coordinates and uncertainties of each point, in az/el. It can only be run after both columns.make_az_el() and points.mark_point().