Least Squared Error Data Reduction

After attending a couple of NARAM's and helping host one, I had plenty of opportunity to see how altitude data was handled. I wasn't very impressed with the add-hoc method used to combine data from more than two tracking stations so tried to think of something that might be better. What I came up with is a least squared error technique.

The method approved in the NAR Sporting Code (a.k.a. the "Pink Book") is to average together the results which close. This is a simple method but it completely ignores the quality of the different results..

The geodesic data reduction method finds the line with minimum length that is perpendicular to the vectors from the two tracking stations. The altitude of the midpoint of this line is the result and the ratio of the lines length to the altitude is the closure figure of merit. (FOM) This is usually called the error but is no such thing. It is just an easily computed metric.

I extend this by finding the point which minimizes the sum of the squared distances from a point to each of the tracking vectors. This is a Least Squared Error (LSE) method. In the two station case, this produces the exact same result as the geodesic method.

For a closure figure of merit I take a different approach. I could use a ratio of the sum of the squared errors to the altitude but I decided that using the angles between the tracking station vectors and the solution point made better sense.

One advantage of this LSE method is that I must locate each tracking station within a 3D cartesian coordinate system. It is therefore trivial to use stations at different absolute altitudes.

To test this method I decided to do two basic tests: Process old contest results to compare the two methods. And apply random error to known exact azimuth and elevation angles and look at the variance in the results.

Here is part of that first test. This table has the E Eggloft Altitude results from NARAM-44. Included are the raw tracking data. (azimuth and elevation) Note that values of '-99' indicate a track lost. (I removed all cases where three or more stations reported track lost.) Then there are the altitude (altx) and closure (clx) numbers from the geodesic method. Values of 0 and 100 indicate that one of the two stations reported a track lost. Then there is the averaged result. The results from my method give the (x,y,z) location of the solution point. I also compute the angular error between each station (alpha, bravo, charlie, delta) and this point. Any angle greater than 5 degrees indicates that this station was not used in the solution. I finish up by computing the percentage difference between the two methods.

That is a lot of data to look at!

(The curious can read the C source.)

Surprisingly, a large number of the angles are around 1 degree or less. Given the nature of the equipment used, this really amazes me. The people working the tracking stations at NARAM are good!

For the most part, the difference between the two methods is slight. So it is hard to say from this which method is better. The Monte-Carlo simulation results will be needed to differentiate the two methods. But by looking at the times the two methods disagree, we still might learn something.

Here is one track where the difference is almost 10%.

Station 2 reported 'track lost' so there are only three possible geodesic solutions. Of these only two close at 347.8m (3.4%) and 452.6m (1.1%). Both solutions indicate that they closed but they produce wildly different results. The LSE results show that all three of the valid stations were kept in the solution with angles of 2.0, 3.8, and 4.4 degrees. I had the program set with a limit of 5 degrees before throwing out a station. If I drop that down to 4 degrees, the station 4 drops out. This results in angles for the two remaining stations of less than 1 degree. Station 4 ends up with and angle of 7 degrees so it looks it was pointing off in the wrong direction.

This shows that the geodesic method can produce results that look good but are obviously not. Where the LSE method can recognize this problem and take care of it. At least when a criteria of 4 degrees error or less is used.

Here is another track where the difference is almost 20%.

All four stations reported but only one of the six possible geodesic solutions closed. And with only a closure figure of 8.5% at that. The LSE method used three of the four stations to arrive at its solution.

Part of the problem here is the very low altitude. The final altitude is under 100 meters and the baselines are all much longer than that. So the distance from the tracking stations was relatively large. The length of the line between the tracking vectors depends on the angular error and the distance from the stations to the track point. This varies only slightly with altitude. But since the geodesic closure figure is the ratio of this length to altitude, low flights are penalized with poor closure figures.

This case has a difference of 5.1%.

One station reported track lost leaving three possible geodesic solutions. Only two of these closed. The two closed values were 475m (3.1%) and 393m (1.6%). Both closure figures indicate high confidence in the solutions. yet they differ by almost 80 m. They both can't be correct. The LSE method used all three stations in its solution with angles of 1.0, 3.2, and 3.7 degrees. So this is a case where the trackers didn't quite agree on where the rocket was resulting in higher than typical variance in the tracking angles. At least with the LSE method you can see this. As to which of the two methods produces the best result it is hard to say. I have my biased opinion but a definitive answer will have to await the Monte-Carlo runs.

One of the interesting side benefits of the LSE method is that it provides a 3D solution rather than just an altitude. This lets you plot the location of all of the flights.

For the curious, the tracking stations at NARAM-44 had the following (x,y) positions:

One of the really bizarre results I found in the NARAM 46 data was the following:

Only three tracking stations arranged in a triangle were used at NARAM 46. The fourth station was not setup because a suitable location at the same elevation as the other three could not be found. Positive angles were to the inside of the triangle so obviously station 2 is pointing outside of that triangle. In spite of this obvious problem, Contest Manager reports a closed track using it. The result is an altitude that is way off. If you look at the angular error from the LSE solution, two stations are spot on while station 2 is off by 25 degrees. If you go with the assumption that the station 2 azimuth should have positive 28 degrees instead you do get a track not closed result but now the other station does close with station 2 but is once again very different at 453 meters.

Monte Carlo runs

In order to better compare the results of the various data reduction methods, I wanted to see how they performed in the presence of random noise. Since I don;t have a good feel for the probability distribution function of real data, I will simply use nice zero mean Gaussian noise. The standard distribution for this noise is 1 degree which seems to be about the right amount to simulate the magnitude of typical errors.

So. pick a point in space and then compute the azimuth and elevation angles for each tracking station. Now added random noise to each of these and round to the nearest half degree (simulating the typical resolution of tracking hardware) and compute the results. Repeat.

After a few thousand runs, compute the standard deviation of the percentage error. To get a feel for things, I wrote the program to use a fixed altitude but to cover the area around the tracking stations with a grid. I then plotted the results using Gnuplot. This was interesting if not very informative until I figured out how to add contours. For now all I have is code to do this for the two station case. Getting all of the angles right to keep the geodesic method happy is giving me fits for more than two stations at the moment. For the two station case, LSE and geodesic (and John DeMar's HIT method as well) produce essentially the same results. So I will put only one plot here:

In this plot, the tracking stations are at x=0, y=0 and x=1000, y=0. Target altitude was 2000. While the error doesn't change much over a very large volume, there is still a definite "sweet spot" on either side of the baseline.

Here are the error plots using the tracking station setup we had for NARAM-44. These are run for altitudes of 300, 600, and 1200 meters. I am putting them side by side to make comparison easier. But if you are stuck with a narrow monitor it might not work too well. Least squared error method on the left and averaged geodesic on the right.

While the least squared error method does have distinctly lower variance than the averaged geodesic method, it isn't a big advantage.

That was the standard deviation of the error, here are some plots of the the average error. The noise in the input data is zero mean. It would be nice if the resulting altitude errors were zero mean but because the equations are highly non-linear, it isn't very likely.

From this, it appears that the performance of the averaged geodesic method is quite good and the least squared method doesn't provide much of an improvement.

In analyzing the NARAM-44 data, I looked at the angle errors for the least squared solution. I broke these down into azimuth and elevation errors. The azimuth error standard deviation was about twice that for the elevation. So what happens to the Monte-Carlo results if the azimuth noise is bumped to a standard deviation of 2 degrees? Once again, least squares is on the left and geodesic on the right.

That was interesting. I had to restrict the range for the geodesic data because it broke down at the edges of the test region and produced very large numbers for the average error and standard deviation. This also messed up the contours. I would have to dig into the results to be sure (but I have no desire to dig into thousands of lines of numbers) but this is probably due to none of the possible solutions closing. The least squared method remains well behaved and again provides better performance than the geodesic method.

Just for grins, I ran the program again but with the four stations placed how it used to be done: in pairs at the end of a 600 meter baseline. Here are the results for a target altitude of 600 meters.

Which isn't very different. The advantage in putting the tracking stations in a rectangular layout is in increasing the chances of at least two trackers seeing the rocket. If you have two trackers at one spot, if one has trouble with the sun, so will the other.

Home