Single-line batch geocoding with ArcGIS Server 10 geocode services

ArcGIS Server 10 rolled out support for single-line geocoding via Web services, which made it easier to provide a simple, search-engine experience when locating an address. While support for single-line geocoding was included with the REST and SOAP APIs, there remained some differences in functionality. For example, REST does not currently provide an operation that supports batching multiple address inputs in a single request. SOAP supports batching via the GeocodeAddresses operation, but it appears to only support multi-line addresses. In fact, batching multiple single-line address inputs is possible through a "hidden" feature. Simply associate the single-line text with the first address input field. For example, in the North American geocode service hosted by ArcGIS Online, the first address input field is named "Address". The GeocodeAddresses operation needs an address table and address input field mapping. The address input field mapping merely references the first address input field, in this example "Address". The address table includes a field that maps to "Address", and the field values contain the single-line address input text. Here's an example in .NET that uses the precompiled ArcGIS SOAP library available with the ArcGIS SOAP SDK:



Note that the suggested batch size of this service is 10, which will limit the number of inputs you can include in one request. And of course you'll want to peruse the terms of use. Links to information on the ArcGIS Server 10 geocode services hosted by ArcGIS Online are available below:

North American Address Locator
United States Street Locator

Get current user info in a Server Object Extension

Server Object Extensions (SOEs) give developers the ability to utilize the extensive geoanalytical capabilities of ArcObjects within an ArcGIS Server service. They also provide a framework that enables access to custom functionality in a service through the standard Web service protocols SOAP and REST. Enabling Web access to an SOE is well documented in the ArcObjects SDK.

ArcGIS Server Web services can also be secured using token or Web server authentication methods. Since SOEs are can be enabled on a specific service, authentication rules configured on a service apply to all of its extensions, including custom SOEs. This poses the question: if an SOE is enabled on a secured Web service, can the authenticated user be retrieved within the SOE? Yes, in ArcGIS Server 10 this is possible.

Use the C# code provided below as an example. Basically, within your SOE code get the current server environment and cast to IServerEnvironment2 to gain access to UserInfo. If authentication is enabled for ArcGIS Server Web services, each request to a service will be associated with an authenticated user. UserInfo will contain the user name and roles within which that user is present for the current request. Note, when Web server authentication is based on Windows users and groups, the user name is in the format domain\user.



Unfortunately, REST SOE's hosted on ArcGIS Server for .NET require ArcGIS Server 10 service pack 2 for this to function. ArcGIS Server 10 service pack 2 will be available for download on the ArcGIS Resource Center soon.

Using Google Maps with ArcGIS Silverlight

The ArcGIS API for Microsoft Silverlight/WPF (and Windows Phone) includes an extensible framework for adding geographic data from a variety of sources. The core library (ESRI.ArcGIS.Client) contains the base classes for the framework and the implementation for the primary source of data, ArcGIS Server. In the initial version of the API, the framework was utilized to support the use of Bing Maps imagery. In version 2.0, the framework was leveraged to support other data sources, such as OpenStreetMap, WMS, and GeoRSS in the toolkit data sources assembly (ESRI.ArcGIS.Client.Toolkit.DataSources.dll) . You'll notice Google is missing from that list of data sources. This brings up a question, and the purpose for this blog post:

Can you use Google Maps imagery in an ArcGIS Silverlight/WPF/Windows Phone application?

There are two answers; one technical, one legal. First, technically it can be done. The Web Mercator projection and the tiling scheme used by Google map and image tiles is virtually the same as Bing, new ArcGIS Online services, and OpenStreetMap. The URL format to access Google tiles directly can be discovered. For example, you can use Fiddler to view requests from a legitimate Google Maps application. Some folks even discuss the URL format and structure (e.g. Code Project).

Now for the legal answer. In short, no. You cannot access Google Maps imagery outside of an interface (read: APIs) provided by Google. This is mentioned in an online FAQ which references an item in the terms of service. The last statement in the terms of service appears to suggest that direct access to map tiles outside of an API is possible via an explicit agreement. In an email conversation with Thor Mitchell, Product Manager for the Google Maps API, he clarified these terms by stating that such agreements are rare and "they are generally limited to embedded device partnerships such as in-car navigation systems and in-flight entertainment systems." Thor also reminded me that the URL format to access Google tiles is an undocumented interface, so it can change at any time. And Google maintains a team that identifies and contacts application developers that use Google tiles (and services) in an unsupported way. If you try, you will likely be asked to stop, followed by more punitive measures if ignored.

So why doesn't Google allow direct access to map tiles?

There are a number of reasons. According to Google, enabling direct access prevents them from meeting their financial obligations to companies from which they license data. Also, there has been no compelling business justification given the risk of abuse. If you'd like to comment on the prospect of direct access to Google map tiles, Google's own Pamela Fox has published an issue on the topic.

So how do you legally use Google Maps data within an ArcGIS Silverlight application?

You'll need to use the Google Static Maps API. Essentially it generates snapshots of Google map tiles at an extent, image size, and scale level that matches the Google Maps tiling scheme. You can think of it as sort of a dynamic tiled layer.

Since the custom layer will generate dynamic map images from a service, it can extend the DynamicMapServiceLayer class in the ArcGIS Silverlight API. You can override the Initialize method to set a few key properties, such as the full extent and spatial reference, but you'll also want to define the tiling scheme that matches Google Maps and will be used to define the scale level at which a dynamic image will be generated. These levels of detail (lods) will be used to determine the output size of the map image generated by the Static Maps API. The primary method to override in this case is GetUrl() which provides access to the map extent, pixel width and height, and a delegate to call when URL construction is complete and a request for a new map image should be generated. At runtime, each extent change in the Map control calls GetUrl() to generate a new map image.

The URL to generate a Google Static Map requires a center point in geographic coordinates, a scale level, and the image size in pixels. The Static Maps API does not support projection on the fly, so the spatial reference will always be Web Mercator. This means the spatial reference of the ArcGIS Silverlight Map control must also be set to Web Mercator. So the GetUrl() method in the custom layer is provided a map extent in Web Mercator. Fortunately the ArcGIS Silverlight API includes a client-side static class, ESRI.ArcGIS.Client.Projection.WebMercator, which can be used to tranform geometry between Web Mercator and Geographic (WGS84) coordinate systems. This can be used to tranform the center point from Web Mercator to geographic coordinates.

The appropriate scale level is determined by matching the resolution (map units per pixel) of the Map with the level of detail in the Google Maps tiling scheme. If they match, the image size passed to the GetUrl() method does not need to change. If they do not match then the image size must be modified to account for the discrepancy. Unfortunately with the Static Maps API the image size is limited to 640x640 pixels. So while the layer will work in between scale levels, you will be able to generate larger images (up to 640 pixels on a side) more reliably if you snap to levels of the Google Maps tiling scheme. Since you cannot associate a tiling scheme with a Map control outside of adding a tiled layer, you must first add a "dummy" tiled layer with the same levels of detail used by Google Maps. The sample download below contains one such layer, MercatorSchemaTiledLayer. Be sure to set the dummy layer's visibility to false (since you'll be using the Static Map API to generate your basemap) and set the SkipToLevels property on the Map control to true.

The screen shot below shows the Google Static Maps API in action. A Static Map is used as a basemap for an ArcGIS Server dynamic map service. Note the Google logo and copyright text at the bottom of the map.

There are a few legal implications to keep in mind when using the Static Maps API:

• The application must run in a Web browser (see 10.8). This means you cannot use it in a WPF, Silverlight out-of-browser, or Windows Phone application.
• All branding and attribution must remain visible at all times, and can not be obscured by overlays or UI elements in any way (see 7.4d). Although the Google copyright text and ESRI logo are close in an ArcGIS Silverlight application, you can still see\read the Google copyright details.
• The application should not attempt to stitch multiple static map images together to display a map that is larger than permitted in the Maps APIs Documentation (see 10.2). Basically you can't generate a bunch of dynamic images and stitch them together on the client for map sizes greater than 640 pixels on a side. Even if you could legally, you'll get a Google stamp on each image, which might show up in the middle of your map - it could get messy.

Download the sample solution to see how to work with the Google Maps Static API in ArcGIS Silverlight.

Special thanks to Thor Mitchell, Product Manager for the Google Maps API, for providing detailed information on the legal use of Google Maps data and APIs.

_

Using the SQL Server Spatial library in Windows Azure

Microsoft recently announced that SQL Azure will support working with native spatial data in June of this year. This is great news, and significantly enhances the usability of SQL Azure as a geographic data respository. As a result, many folks may want to start utilizing spatial data in their Azure hosted Web applications. Working with SQL Server spatial data in a .NET application usually involves working with the SQL Server Spatial (SqlServerSpatial.dll) and Types (Microsoft.SqlServer.Types.dll) libraries included with the SQL Server System CLR Types feature pack. The technical capabilities of the SQL Server Spatial library have been established in numerous blog posts, conference sessions, forums, and in the product documentation. Basically the library offers a standard, light-weight set of spatial operators built on geometry and geography spatial data types that are relatively easy to use and integrate in a .NET application. You can download, install, use, and distribute these libraries as needed. The latest edition of the SQL CLR Types was released in November 2009 for SQL Server 2008 R2. If the .NET application that uses the SQL spatial libraries is installed on a local machine or hosted on a local server, you have complete control over the environment in which it operates. However, if you choose to deploy an ASP.NET application as a Windows Azure web service, you will encounter a limitation. The platform for Windows Azure is a flavor of 64-bit Windows Server 2008 prior to the R2 release. The Nov 2009 SQL Spatial library is dependent on the system library MSVCRT80.dll which is not included with this operating system. As a result, when you attempt to use the SQL Spatial library in your Azure hosted ASP.NET web service, you'll likely see the following error:

Unable to load DLL 'SqlServerSpatial.dll': The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail. (Exception from HRESULT: 0x800736B1)

MSVCR80.dll is required by the SQL Spatial library and must be installed/configured on the host operation system, it cannot simply be dropped in the same directory. In addition, you cannot install features and components in your Windows Azure workspace. So how can you use the SQL Spatial library in an Azure hosted application? Simple, include a SQL Spatial library from a SQL Server 2008 (not R2) edition of the CLR Types. The functionality is basically unchanged.

Here are the steps:

1) Download the 64-bit SQL Server 2008 edition of the SQL Server CLR Types feature pack. I used the October 2008 download successfully.

2) If you don't want to install the feature pack, or you don’t have a 64-bit system on which to install, you can extract the 64-bit version of this library using the following command:

msiexec /a SQLSysClrTypes_64bit.msi /qb TARGETDIR="C:\MyFolder"

3) Copy the Microsoft.SqlServer.Types.dll and SqlServerSpatial.dll into the bin of your Web application. I was able to successfully deploy and use SQL Server Spatial logic in Windows Azure service using the February 2010 edition of the Azure SDK.

Now let's change gears and chat about ESRI's foray into the SQL Server Spatial and Azure world. Currently ESRI has a product named MapIt which includes an ASP.NET Web service, the Spatial Data Service, that uses the SQL Server Spatial library to work with native spatial data in SQL Server 2008. The Spatial Data Service can be deployed as a Windows Azure service and work with spatial data in SQL Azure or any third party that hosts spatial data in SQL Server 2008 (e.g. Discount ASP.NET). You can see the SDS in Windows Azure in action here: http://mapit.cloudapp.net/. To deploy your own SDS as a Windows Azure service, use the Azure deployment utility on the MapIt resource center.

As a parting thought, keep in mind that Microsoft will likely update the Windows Azure platform to include the foundational libraries necessary to support the SQL Server 2008 R2 edition of the spatial library. Hopefully the information in this post will suffice until that time.

Dev Summit 2010, Image Services, and Silverlight

About a week before the Dev Summit I was tasked with showcasing ArcGIS Server 10 image services in the plenary using Silverlight as the platform. The tricky part was limiting the temptation to “light it up” with Silverlight so as not to overshadow image services. And there was another problem - the last time I did anything related to image processing my cell phone was the size of an army field radio, Netscape was the browser du jour, and Enron was a hot stock buy. So I had to ramp up on image services quickly… which meant spending some quality time with the raster gurus at the factory (ESRI). Along the way, I gathered some valuable info and thought it might be interesting to relay a few pivotal gems on the use of image services and Silverlight in ArcGIS 10.

• ArcGIS 10 supports a new geodatabase model, the mosaic dataset, which can easily and effectively manage small and large collections of imagery. They are created and maintained in ArcMap or ArcCatalog and you use a set of GP tools to manage. Mosaic datasets support imagery from multiple sensor platforms including QuickBird, IKONOS, and WorldView. Most importantly, there is no preprocessing required. Mosaicking, pansharpening, and ortho-rectification happen on the fly. Merely add the image to the mosaic dataset and it assimilates.
  
  
• Mosaic datasets support time awareness. Image services are time aware if the underlying mosaic dataset has been time enabled. You enable time awareness in the mosaic dataset properties, not the layer properties (as with feature layers in a map service).
  
  
• Sometimes rasters in a mosaic dataset will not display at full or larger extents. In this case, overviews must be generated. Overviews are similar to pyramid layers in that they allow for the quick display of very large collections of imagery. Overviews do this by resampling source data to generate "snapshot" images for larger map resolutions (larger extents). In most cases, you will want to generate overviews. However, this poses a problem when you want to display a set of rasters that occupy the same space in a time series. Overviews are generated for a specific map resolution (map units/pixel) ranges, not a specific instant or span of time. Therefore if you're viewing your image service at a map resolution that uses an overview, the overview will only show one instant of time regardless of the time extent you specify. In this situation, avoid generating overviews and increase the maximum cell size range factor to enable the display of primary rasters at larger map resolutions. To do this, use the Calculate Cell Size Ranges option from the mosaic dataset context menu in ArcCatalog/ArcMap or use the geoprocessing tool of the same name. Increase the range factor to increase the threshold at which overviews are used, if available.
  
  
• 9.3.1 Image Server functionality is now part of core ArcGIS Server 10.
  
  
• All rasters, including mosaic datasets, can be published as image services. An individual image on disk is termed a "raster dataset" or "raster layer".
  


• Image services include an enhanced export image operation and three new operations: query, identify, and download. Each are covered in detail below.
  


• The export image operation has a new JPGPNG image format which lets the image service define the ideal output format. If the extent requested contains transparent portions, generate a png. If not, generate a jpg.
  


• The export image operation uses mosaic rules to order, filter, and select tiles displayed in the mosaicked image. For example, if a service hosts a set of overlapping images you can define which ones are displayed over others based on an attribute such as date, or a point of interest. The REST API documentation provides minimal insight into parameter descriptions. The SOAP API documentation may provide more information (it definitely will in ArcGIS Server 10 final). Also note, the identify operation uses mosaic rules to order and filter rasters to modify results.
  


• The export image operation provides rendering rules to dynamically change the display of raster data using a set of predefined raster functions. For example, you can apply a custom aspect, slope, color map or shaded relief on the fly. Only 8 raster functions are available with the ArcGIS Server 10 REST API. One item you may notice in a rendering rule is the variableName attribute. It must be set to the value defined in the REST API doc, either “DEM” or “Raster”. It represents a property defined in ArcObjects where values may include “Raster2” or a user defined name for a function chain. These other values are not exposed in the REST API.
  


• The query operation enables clients to define an attribute and/or spatial query and return information about rasters in a dataset hosted by an image service. You can also use it to return raster footprints or a raster thumbnail.
  


• The identify operation enables clients to return a pixel value for rasters hosted in an image service given a user-defined location. The location can be a point or polygon. If a polygon, the centroid of the polygon is used. The order of rasters during the operation can be defined by a mosaic rule. Unlike the identify operation on a map service, image services do not provide a tolerance since only one pixel value per raster can be returned. Do not let the pixel size parameter fool you; it defines a pixel resolution to use when identifying pixel values. This means you can identify on overviews available in the mosaic dataset, and return an interpolated value in an overview image. Interpolation occurs when the overview is generated by resampling source rasters.
  


• The download operation basically enables clients to download individual rasters and metadata hosted by an image service. You can also clip imagery on the server before download and define the download image format. If you don't want users to download individual rasters, you can easily disable this capability of the service.

So what does this mean for the ArcGIS API for Microsoft Silverlight/WPF? To put it simply, the ArcGISImageServiceLayer has been updated to support working with ArcGIS Server 10 image services. A set of new classes will be included to support mosaic and rendering rules. The QueryTask can be used with the image service Query operation. The Identify operation will require a new task, and the Download operation may be supported in the near future. 2.0 public beta will be released in April and you'll get a chance to try out some of this new functionality.

Also, at some point in the near future a set of ArcGIS Server 10 image services will be hosted online by ESRI to demonstrate their functionality. They may or may not be officially announced. In any case, I’ll tweet when they’re up, running, and public (@rex_hansen).

ESRI's European User Conference 2009 - Asides and Slides

In October 2009 HNIT-Baltic hosted ESRI's European Users Conference in Vilnius, Lithuania. I attended as an ambassador for ArcGIS Server, its Web APIs, and MapIt. This was my first foray into eastern Europe, and it turned out quite simply fantastic. Discussions with users and partners were spirited and exciting, and attendees in my sessions were engaged. The folks I met that work at HNIT-Baltic were exceptionally knowledgable and courteous, and simply ready for the next challenge.

The city itself was inviting, fresh, and personal. And I discovered a few things during my adventure: Lithuanians enjoy pizza with ketchup, because I was told it "adds flavor"; shopping malls look like factories on the outside, but they rival the best of So Cal on the inside, with better prices; and if you see a glass of water on a table at a party... it's probably vodka.

I've included a montage of a few photos I snapped with my trusty iPhone while in Vilnius (press "P" to pause).

If you attended the EUC in Vilnius or just want to view the conference materials, you can view and download the plenary and paper session materials from HNIT-Baltic.

To peruse the technical sessions I presented, I've made PDF versions of the session slides available below:
Building Mashups with ArcGIS Server
Building Great Web Map Applications
Whats New in ArcGIS Server 9.3.1

Using the ArcGIS Server REST API in .NET applications

Over the past year I've handled a number of questions regarding the use the ArcGIS Server REST API in a .NET application. It can be done quite effectively, but isn't explicitly promoted by ESRI since the REST API is primarily designed to be consumed by pre-packaged Web clients like the ArcGIS JavaScript, Flex, and Silverlight/WPF APIs. However, using REST programmatically in a .NET application context without installing an ESRI product can make for some powerful and focused yet lightweight solutions. In addition, if you need to do something relatively simple you don't have to generate a load of client-side proxy classes like you do with SOAP.

I've included a simple code example below to highlight how easy it is to integrate ArcGIS Server services via REST in your .NET app. The example uses an ArcGIS Online geocode service to geocode an address and return a location.

Download the sample here.

There are a few items to consider when using ArcGIS services via REST in .NET:

1. You will construct the url that will execute an operation on the server. Inputs to the operation are included in the url as parameters with values. Add a reference to the System.Web.dll and use the HttpUtility class to encode parameter values in the url.
2. In most cases you will return results in JSON (JavaScript Object Notation) format. Add a reference to the System.Web.Extensions.dll (included with .NET 3.5, separate install for .NET 2.0) and use the JavaScriptSerializer class to deserialize JSON results into a dictionary you can use more effectively.

This example can easily be modified for use in a .NET CLR trigger for SQL Server to geocode address information upon entry into a table -or- in a batch process to geocode addresses in a database -or- show a location in a WinForm or WPF desktop client. The same idea can cross over into other ArcGIS Server services such as querying layers in a map or executing a geoprocessing service. In any case, you'll find the simplicity of the REST API refreshingly simple for those scenarios where you need to plug a little GIS into your app. One parting note, if you use ArcGIS Online services via REST, you'll need to follow the subscription model for the Web Mapping APIs. For example if you geocode more than 1000 addresses, you'll need to purchase a subscription.