Killer Data

Back in 2005 I wrote two article on LiDAR. These articles are:

Killer Apps – Part 1

(Here is a text version of the article.)

I have been musing about some threads relating to data for a while and I’ve noticed they have been tangling people up.These threads resonated at the recent Spatial Sciences Institute (SSI) Conference held in Melbourne, Sept.12th-16th.I want to unravel a few threads in light of that conference.

It has struck me for quite a while how crazy things are in our industry.Let me give an analogy that describes, until recently the bind in which we found ourselves.Your hi-fi system dies.Off to the hi-fi shop you go and, before you know it, you are walking out of the shop with a system that would take someone with a Ph.D.to use! Needless to say, your wallet is very much lighter.Your new system, you tell your friends, is the best – it has features, features and more features.Then the awful truth dawns on you: you can’t afford the music CDs!

I have heard lots and lots of people tell me how they have the biggest and best GIS around.(What my previous employer’s CEO called a “Rolls-Royce GIS.”) But then they go on to bemoan the fact that they can neither access nor afford the data.

In the Australian environment the question of data access and pricing is a perennial one.So it was no surprise that it was talked about constantly at the recent GITA Australia (reviewed here) and SSI conferences.

Stuart Nixon: The “Where” of Data
Stuart Nixon, CEO of ERMapper (based in Perth, Western Australia,) gave a keynote presentation on the first day of the SSI conference.He provided a marketing/financial view on what, in his view, is holding back the geospatial industry.

Nixon explained how he thought we were good at capturing data; pretty good at managing it (though I beg to differ there); but pretty awful at delivery.By “delivery,” he wasn’t referring to its technical aspects per se, rather he meant the way in which we put our services in the hands of the masses.In his view the latter needs to change, and change quickly.To create a framework for discussing it, he characterized the spatial market as being:

1. Small but expert;
2. Something that, when compared to search engines, is not used by everyone;
3. Restricted by business and data copyright models;
4. Characterized by poorly integrated data; and
5. Distinguished by having most of its data locked up.

In Nixon’s view, accessibility is the issue.

“What” and “Where”
To illustrate, Nixon compared the value of the “What” industry (i.e. search engines like Google, MSN, Yahoo!, etc.) to the “Where” industry (us).The “What” industry, in his view, generates money, whereas the “Where” industry spends it.The “What” industry does this by giving access to information about things into the hands of anyone.This is the value of “What” – accessibility.Nixon presented a diagram that he used to compare the two industries.

Advertising is an area in which the “What” industry makes its money, and is the one that our industry needs to work on to turn it from being a cost sink to a revenue generator.He also thinks that “GeoLocation,” i.e., geocoding services, is the area stopping the industry from generating revenue in the advertising space.He stated quite emphatically that “geocoding does not exist.” So Nixon believes that the value of “Where” – on which our industry’s future lies – has not been addressed.

A Way Forward?
Nixon thinks there is a way forward but first three missing things need to be fixed.

1. We can’t locate (geocode) things. Geocoding for Nixon is the equivalent of a Google “search.” He thinks there is no unified approach and that it is costly (currently about one cent per transaction).Nixon stated that the industry must stop seeing geocoding as the business model but rather as the enabling technology for value-added services that anyone can use.Part of the issue, Nixon believes, is the lack of free street databases on which to base geocoding.

I take a somewhat different view that Nixon on this. I simply don’t believe that geocoding is a geospatial process.My view is that it is a semantic matching problem in which two fuzzy sentences (addresses) are matched for the purpose of transferring two attributes: a latitude and a longitude.Yes, geocoding requires some geoprocessing to create a matchable dataset (from multiple spatial sources) but it does not need a GIS to execute a geocode operation.

2. Current data licensing models and access modes are too restrictive. Nixon highlighted the dilemma when asked the rhetorical question: Why is our satellite imagery rotting in tape libraries when it should be getting used? He also underlined his passion on this topic when he proclaimed that there is better, free, detailed coverage of Mars than of Earth! Sure, Australia has some large datasets, but current license and access agreements are killing access.

Part of the problem is that the major suppliers of data are the government agencies.Some of the data are free but some are not.The problem is compounded by the fact that much of the data (e.g. topographic data) was captured in the days before economic rationalism saw governments retreat from major infrastructure projects like mapping.

This is not an issue that is isolated to Australia. In my view some of the current owners of navigable road datasets will face this dilemma if they continue to price themselves out of Internet portal use.The restrictions on use of road data inside Google Maps are not Google’s fault.It is the data licensing that is the issue: this licensing runs counter to the portal’s approach of ubiquitous access.If this becomes a problem, Google and the other location enabled portals just might go elsewhere (or capture it themselves).

I think new technologies like LiDAR (Light Detection And Ranging, discussed below and defined here) have the potential to change this. Much more on this in Part 2 of this article, which will appear next week.

3. Current data delivery mechanisms are poor. Those of us who thought CD and DVD technologies were a boon to data delivery can think again.Now even these are not good enough.In the Internet age, the main method of delivery should be via the Web (for those who can afford the bandwidth).

Putting Money Where Mouth Is!
Nixon drove home his point that “usage equates to value” when he made a special announcement that ERMapper, in conjunction with the Australian Greenhouse Office and GeoScience Australia is sponsoring a new web site to enable greater access to large geospatial datasets.

Before Nixon made the announcement, he noted that BitTorrent traffic takes up to 35% of today’s Internet traffic and is, therefore, the delivery mechanism of choice for Internet data delivery.The new site Nixon announced is called www.geoTorrent.org.

The technological vision and driver for the new site is one of ERMapper’s staff, Richard Orchard.It uses BitTorrent technology to improve user access and delivery, thereby hoping to increase usage and value.It is not meant for online access or searching, rather simply for data delivery.The great thing about this site is that the last 30 years of Landsat imagery for Australia is now available at one access point! Perhaps not “killer data,” but it is a great start that will hopefully cause others to open their data vaults to the public.

Too late?
But is this a late response from an industry caught with its pants down by Google? Is it already game set and match to Google?

Nearly.An example may help clarify why I think this might be the case. In my home state, Tasmania, there has been a ground swell of demand for high resolution imagery for use in the private and public sectors. Local government (of which there are 29 councils in Tasmania), fire, police, ambulance, parks and other state government agencies, forestry companies, agribusinesses, etc., all want this data for their many needs.None of them, as individuals, can afford to buy and process imagery from the main suppliers to create a single seamless mosaic over their operational areas and keep it up-to-date.But, as a single body, they could afford such imagery.

The problem is that the main agency for coordinating data capture and access within the state, The LIST, has not been responsive to this demand.It has not gathered the many players together to ascertain need (coverage, scale, sensors, time scales), draw up common specifications, negotiate with suppliers, and then coordinate the processing of the product to make it available to all.Something might be happening as I write, but this is a reactive response rather than proactive service provision.

At a recent local Tasmanian conference, I publicly asked the question: “Do we have to wait until Google buys all the DigitalGlobe data for Tasmania, processes it and makes it available in Google Maps/Google Earth before we can finally get the data we need?” Someone in the audience yelled out: “Too right, mate!” If my listener’s response is any measure, the decision has been made: we will wait for Google to do what we seem incapable of doing ourselves!

What Google has done in this space can be likened to “killer data.” In my Killer Apps article published last week, I used the term in the following sense (from Wikipedia): “The term is especially used when the technology existed before but did not take off before the introduction of the killer application.” We know that large data stores existed before Google Maps and Google Earth, but these were not as accessible or useful (raw data for download is not what most people want).Thus, it seems to me that the only real data integration Web Service that has any traction in the market today is Google’s KML (reviewed here). One attendee at the SSI conference informed me that Google does have a contract with DigitalGlobe for their data and is buying up an incredible amount of it.

But with Google, geospatial data’s presence has become visible and so is now “taking off,” as it were.Google knows data is everything in the geospatial business.One attendee at the SSI conference informed me that Google has a contract with DigitalGlobe for their data and is buying an incredible amount.This data will be made accessible via its platform interfaces (Maps API and KML).Perhaps it is these interfaces that will open up data access far more than the ones the geospatial industry has been pushing for quite a while now.

In summary, let’s keep it simple.If I could provide my users a Google Maps layer within GIS application(s) in the same way we access WMS services, then it really is close to game, set and match! Tasmanian users will get what they want: seamless, statewide coverage of up-to-date high resolution imagery.And, just in case Google is listening, they would be willing to pay!

Enjoy.

Killer Data – Part 2

(The article is no longer available. Here is a text version of the article.)

LiDAR
But satellite imagery, no matter the resolution, will not cover all our needs. There is also, to my current thinking, a non-satellite based technology that has the potential to provide up-to-date, accurate data where previously we had to use statistical sampling and costly field survey and ground-truth programs to capture data: LiDAR. (Light Detection And Ranging, defined here.) I believe that LiDAR technology has the potential to be one of the next big killer data sources. Again, let me explain using two examples.

Forestry. Much of the GIS data in Australia is inaccurate, old and so very out-of-date. Much of the data was captured for topographic mapping purposes and dates back over 30 years. In remote, heavily forested areas, geodetic control was poor and photogrammetrists struggled to identify spot-heights and stream networks through heavy canopy.

Foresters have a need for accurate digital elevation data (particularly for cable harvesting, a method of removing trees rather like a cable car at a ski resort – you put stations at key points and then run cables between them to pick up the logs once the trees are cut down, and landscape visualization, etc.) and drainage networks (to determine stream side reserves for statutory compliance when harvesting, to determine water catchments, etc). They also need accurate road networks (for coupe-to-mill logistics, road design and cost control). But the main killer data that foresters, specializing in native forestry, have needed for years are data that would help them accurately measure the standing volume of timber in a forest, using as much automation as possible.

Satellite imagery has been tracked for years as a potential source for this but, at least in Australian native forestry, these data were not of much use. In Tasmania, current techniques involve manual photogrammetry – to delineate stands of trees of similar height and density – coupled with unbiased statistical sampling, field survey and predictive computer modeling.

A recent LiDAR trial in Tasmania showed just how much LiDAR is threatening to be killer data. I saw a forester process the results of a trial of around 60,000ha of forest using the open source System for Automated Geoscientific Analyses (SAGA) and the powerful Manifold GIS ($245). First, he took the LiDAR first return (the “first thing” the laser altimeter hits, which could be a top of a tree, for example) and ground points (the actual ground, lower than a tree top), then he Kriged (a statistical process that fills in between the laser points to create a continuous surface) both as separate surfaces, subtracted them and then ran a simple tree crown algorithm to find individual trees. He then went on to demonstrate how he could predict the best trees for thinning, find the old-growth trees, etc. This was awesome stuff! He also took the Kriged ground surface and produced an accurate drainage network from it. From the ground surface a set of nested catchments could also be produced.

This fellow was an inventory forester in charge of the unbiased statistical sampling measurement program. He was so excited by what he had been able to do that it started me thinking that LiDAR just might be the next “killer data” data source!

Sure, LiDAR still measures the world by sampling, but its power is that its sampling can be tuned to the objects being measured, which combined with carefully designed post-processing, enables us to get very close to measuring everything! With LiDAR, my forester friend came close to an estimate of actual trees and their properties (height and density) in a forest whereas previously he was left to extrapolate an estimate from statistical samples.

Roads and Other Infrastructure Data
A digital elevation model (DEM) lets us get at drainage, a powerful piece of infrastructure data that can be extracted from LiDAR. What else is there?

We could identify all the roads in the sample area. We could see the pavement area and, with suitable hyperspectral scanners, probably measure road surface and condition as well. But could we extract accurate 3D road centerlines from the LiDAR data just as we can extract drainage lines from the DEM?

At the SSI conference in Melbourne I found that Merrick now has an Australian connection via Merrick Mars LiDAR of Brisbane, Queensland, Australia. At the stand I had some time to “chew the fat” with Gary Outlaw, Merrick’s US-based VP for business development, and had a look at their MARS software. The conversation confirmed what I suspected – it is possible to extract a 3D road centerline from the same LiDAR data my forester friend was extracting products specific to forestry.

Having a 3D, accurate road centerline dataset would be of immense value to the state of Tasmania. Transportation logistics could be computed from real road slope distances rather than the planimetric distances currently being used. Transportation models could take into account the hilly nature of Tasmania, and model actual truck behavior (e.g., slow speeds while ascending a hill or descending under airbrakes). To these data we could then add statutory road speeds, traffic lighting, one-way attribution, etc., to get a truly useful dataset that would drive economic efficiency.

Data as Foundational Infrastructure
So, let’s return to where this reflection on data started. Put bluntly, the community wants better, up-to-date, data. Google is showing us the power of data and, as Nixon points out, the power of the value-add! The current out-of-date data infrastructure is simply not capable of being the powerful, accurate foundation on which a modern economy can innovate and value-add.

Government departments, the custodians of the existing topographic data, have retreated from significant and active data capture programs due, in part, to “economic rationalism.” With little money to capture newer data, these departments spend what little resources they have managing out-of-date topographic and thematic data. So, in Tasmania we are still editing a drainage dataset that, outside of the urban areas, is woefully inaccurate and incomplete. This is also true for digital elevation data; and even extends to our road network. Even where there is an active capture program in place for roads, the turn around time from edit to publication is slow.

LiDAR has the potential to be “killer data” because it can rid us of the costs involved in trying to sample a complex reality and trying improve the impossibly inaccurate. LiDAR can help us capture new and accurate data using automated methods that require relatively little in the way of costly field-capture programs or other manual methodologies. It can liberate operational expenditure from trying to make a silk purse (good data) out of a sow’s ear (bad data), to concentrating on value-adding the large amounts of data that will become available to us via Google (and other mapping portal efforts) and LiDAR. Because, as technology improves and pricing drops (LiDAR), or data use grows exponentially via new uses (Google), we will be swamped with data riches that require us to “think outside the square” of our own industry.

Conclusion
The great thing about the Google and map portal phenomenon is that it is forcing the geospatial community to look at itself long and hard. To use LiDAR and newer data sources effectively, we have to start thinking about the value-add and not the technology per se. To this end, Nixon’s talk was timely and GeoTorrent.org is the response of one geospatial industrial visionary.

What will yours be?