RexHansen.com

Single-line batch geocoding with ArcGIS Server 10 geocode services

2011-07-24T02:33:00.000-07:00

ArcGIS Server 10 rolled out support for single-line geocoding via Web services, which made it easier to provide a simple, search-engine experience when locating an address. While support for single-line geocoding was included with the REST and SOAP APIs, there remained some differences in functionality. For example, REST does not currently provide an operation that supports batching multiple address inputs in a single request. SOAP supports batching via the GeocodeAddresses operation, but it appears to only support multi-line addresses. In fact, batching multiple single-line address inputs is possible through a "hidden" feature. Simply associate the single-line text with the first address input field. For example, in the North American geocode service hosted by ArcGIS Online, the first address input field is named "Address". The GeocodeAddresses operation needs an address table and address input field mapping. The address input field mapping merely references the first address input field, in this example "Address". The address table includes a field that maps to "Address", and the field values contain the single-line address input text. Here's an example in .NET that uses the precompiled ArcGIS SOAP library available with the ArcGIS SOAP SDK:

Note that the suggested batch size of this service is 10, which will limit the number of inputs you can include in one request. And of course you'll want to peruse the terms of use. Links to information on the ArcGIS Server 10 geocode services hosted by ArcGIS Online are available below:

North American Address Locator
United States Street Locator

Get current user info in a Server Object Extension

2011-04-10T00:55:00.000-07:00

Server Object Extensions (SOEs) give developers the ability to utilize the extensive geoanalytical capabilities of ArcObjects within an ArcGIS Server service. They also provide a framework that enables access to custom functionality in a service through the standard Web service protocols SOAP and REST. Enabling Web access to an SOE is well documented in the ArcObjects SDK.

ArcGIS Server Web services can also be secured using token or Web server authentication methods. Since SOEs are can be enabled on a specific service, authentication rules configured on a service apply to all of its extensions, including custom SOEs. This poses the question: if an SOE is enabled on a secured Web service, can the authenticated user be retrieved within the SOE? Yes, in ArcGIS Server 10 this is possible.

Use the C# code provided below as an example. Basically, within your SOE code get the current server environment and cast to IServerEnvironment2 to gain access to UserInfo. If authentication is enabled for ArcGIS Server Web services, each request to a service will be associated with an authenticated user. UserInfo will contain the user name and roles within which that user is present for the current request. Note, when Web server authentication is based on Windows users and groups, the user name is in the format domain\user.

Unfortunately, REST SOE's hosted on ArcGIS Server for .NET require ArcGIS Server 10 service pack 2 for this to function. ArcGIS Server 10 service pack 2 will be available for download on the ArcGIS Resource Center soon.

Using Google Maps with ArcGIS Silverlight

2010-09-14T00:29:00.000-07:00

The ArcGIS API for Microsoft Silverlight/WPF (and Windows Phone) includes an extensible framework for adding geographic data from a variety of sources. The core library (ESRI.ArcGIS.Client) contains the base classes for the framework and the implementation for the primary source of data, ArcGIS Server. In the initial version of the API, the framework was utilized to support the use of Bing Maps imagery. In version 2.0, the framework was leveraged to support other data sources, such as OpenStreetMap, WMS, and GeoRSS in the toolkit data sources assembly (ESRI.ArcGIS.Client.Toolkit.DataSources.dll) . You'll notice Google is missing from that list of data sources. This brings up a question, and the purpose for this blog post:

Can you use Google Maps imagery in an ArcGIS Silverlight/WPF/Windows Phone application?

There are two answers; one technical, one legal. First, technically it can be done. The Web Mercator projection and the tiling scheme used by Google map and image tiles is virtually the same as Bing, new ArcGIS Online services, and OpenStreetMap. The URL format to access Google tiles directly can be discovered. For example, you can use Fiddler to view requests from a legitimate Google Maps application. Some folks even discuss the URL format and structure (e.g. Code Project).

Now for the legal answer. In short, no. You cannot access Google Maps imagery outside of an interface (read: APIs) provided by Google. This is mentioned in an online FAQ which references an item in the terms of service. The last statement in the terms of service appears to suggest that direct access to map tiles outside of an API is possible via an explicit agreement. In an email conversation with Thor Mitchell, Product Manager for the Google Maps API, he clarified these terms by stating that such agreements are rare and "they are generally limited to embedded device partnerships such as in-car navigation systems and in-flight entertainment systems." Thor also reminded me that the URL format to access Google tiles is an undocumented interface, so it can change at any time. And Google maintains a team that identifies and contacts application developers that use Google tiles (and services) in an unsupported way. If you try, you will likely be asked to stop, followed by more punitive measures if ignored.

So why doesn't Google allow direct access to map tiles?

There are a number of reasons. According to Google, enabling direct access prevents them from meeting their financial obligations to companies from which they license data. Also, there has been no compelling business justification given the risk of abuse. If you'd like to comment on the prospect of direct access to Google map tiles, Google's own Pamela Fox has published an issue on the topic.

So how do you legally use Google Maps data within an ArcGIS Silverlight application?

You'll need to use the Google Static Maps API. Essentially it generates snapshots of Google map tiles at an extent, image size, and scale level that matches the Google Maps tiling scheme. You can think of it as sort of a dynamic tiled layer.

Since the custom layer will generate dynamic map images from a service, it can extend the DynamicMapServiceLayer class in the ArcGIS Silverlight API. You can override the Initialize method to set a few key properties, such as the full extent and spatial reference, but you'll also want to define the tiling scheme that matches Google Maps and will be used to define the scale level at which a dynamic image will be generated. These levels of detail (lods) will be used to determine the output size of the map image generated by the Static Maps API. The primary method to override in this case is GetUrl() which provides access to the map extent, pixel width and height, and a delegate to call when URL construction is complete and a request for a new map image should be generated. At runtime, each extent change in the Map control calls GetUrl() to generate a new map image.

The URL to generate a Google Static Map requires a center point in geographic coordinates, a scale level, and the image size in pixels. The Static Maps API does not support projection on the fly, so the spatial reference will always be Web Mercator. This means the spatial reference of the ArcGIS Silverlight Map control must also be set to Web Mercator. So the GetUrl() method in the custom layer is provided a map extent in Web Mercator. Fortunately the ArcGIS Silverlight API includes a client-side static class, ESRI.ArcGIS.Client.Projection.WebMercator, which can be used to tranform geometry between Web Mercator and Geographic (WGS84) coordinate systems. This can be used to tranform the center point from Web Mercator to geographic coordinates.

The appropriate scale level is determined by matching the resolution (map units per pixel) of the Map with the level of detail in the Google Maps tiling scheme. If they match, the image size passed to the GetUrl() method does not need to change. If they do not match then the image size must be modified to account for the discrepancy. Unfortunately with the Static Maps API the image size is limited to 640x640 pixels. So while the layer will work in between scale levels, you will be able to generate larger images (up to 640 pixels on a side) more reliably if you snap to levels of the Google Maps tiling scheme. Since you cannot associate a tiling scheme with a Map control outside of adding a tiled layer, you must first add a "dummy" tiled layer with the same levels of detail used by Google Maps. The sample download below contains one such layer, MercatorSchemaTiledLayer. Be sure to set the dummy layer's visibility to false (since you'll be using the Static Map API to generate your basemap) and set the SkipToLevels property on the Map control to true.

The screen shot below shows the Google Static Maps API in action. A Static Map is used as a basemap for an ArcGIS Server dynamic map service. Note the Google logo and copyright text at the bottom of the map.

There are a few legal implications to keep in mind when using the Static Maps API:

The application must run in a Web browser (see 10.8). This means you cannot use it in a WPF, Silverlight out-of-browser, or Windows Phone application.
All branding and attribution must remain visible at all times, and can not be obscured by overlays or UI elements in any way (see 7.4d). Although the Google copyright text and ESRI logo are close in an ArcGIS Silverlight application, you can still see\read the Google copyright details.
The application should not attempt to stitch multiple static map images together to display a map that is larger than permitted in the Maps APIs Documentation (see 10.2). Basically you can't generate a bunch of dynamic images and stitch them together on the client for map sizes greater than 640 pixels on a side. Even if you could legally, you'll get a Google stamp on each image, which might show up in the middle of your map - it could get messy.

Download the sample solution to see how to work with the Google Maps Static API in ArcGIS Silverlight.

Special thanks to Thor Mitchell, Product Manager for the Google Maps API, for providing detailed information on the legal use of Google Maps data and APIs.

Using the SQL Server Spatial library in Windows Azure

2010-04-30T22:42:00.000-07:00

Microsoft recently announced that SQL Azure will support working with native spatial data in June of this year. This is great news, and significantly enhances the usability of SQL Azure as a geographic data respository. As a result, many folks may want to start utilizing spatial data in their Azure hosted Web applications. Working with SQL Server spatial data in a .NET application usually involves working with the SQL Server Spatial (SqlServerSpatial.dll) and Types (Microsoft.SqlServer.Types.dll) libraries included with the SQL Server System CLR Types feature pack. The technical capabilities of the SQL Server Spatial library have been established in numerous blog posts, conference sessions, forums, and in the product documentation. Basically the library offers a standard, light-weight set of spatial operators built on geometry and geography spatial data types that are relatively easy to use and integrate in a .NET application. You can download, install, use, and distribute these libraries as needed. The latest edition of the SQL CLR Types was released in November 2009 for SQL Server 2008 R2. If the .NET application that uses the SQL spatial libraries is installed on a local machine or hosted on a local server, you have complete control over the environment in which it operates. However, if you choose to deploy an ASP.NET application as a Windows Azure web service, you will encounter a limitation. The platform for Windows Azure is a flavor of 64-bit Windows Server 2008 prior to the R2 release. The Nov 2009 SQL Spatial library is dependent on the system library MSVCRT80.dll which is not included with this operating system. As a result, when you attempt to use the SQL Spatial library in your Azure hosted ASP.NET web service, you'll likely see the following error:

Unable to load DLL 'SqlServerSpatial.dll': The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail. (Exception from HRESULT: 0x800736B1)

MSVCR80.dll is required by the SQL Spatial library and must be installed/configured on the host operation system, it cannot simply be dropped in the same directory. In addition, you cannot install features and components in your Windows Azure workspace. So how can you use the SQL Spatial library in an Azure hosted application? Simple, include a SQL Spatial library from a SQL Server 2008 (not R2) edition of the CLR Types. The functionality is basically unchanged.

Here are the steps:

1) Download the 64-bit SQL Server 2008 edition of the SQL Server CLR Types feature pack. I used the October 2008 download successfully.

2) If you don't want to install the feature pack, or you don’t have a 64-bit system on which to install, you can extract the 64-bit version of this library using the following command:

msiexec /a SQLSysClrTypes_64bit.msi /qb TARGETDIR="C:\MyFolder"

3) Copy the Microsoft.SqlServer.Types.dll and SqlServerSpatial.dll into the bin of your Web application. I was able to successfully deploy and use SQL Server Spatial logic in Windows Azure service using the February 2010 edition of the Azure SDK.

Now let's change gears and chat about ESRI's foray into the SQL Server Spatial and Azure world. Currently ESRI has a product named MapIt which includes an ASP.NET Web service, the Spatial Data Service, that uses the SQL Server Spatial library to work with native spatial data in SQL Server 2008. The Spatial Data Service can be deployed as a Windows Azure service and work with spatial data in SQL Azure or any third party that hosts spatial data in SQL Server 2008 (e.g. Discount ASP.NET). You can see the SDS in Windows Azure in action here: http://mapit.cloudapp.net/. To deploy your own SDS as a Windows Azure service, use the Azure deployment utility on the MapIt resource center.

As a parting thought, keep in mind that Microsoft will likely update the Windows Azure platform to include the foundational libraries necessary to support the SQL Server 2008 R2 edition of the spatial library. Hopefully the information in this post will suffice until that time.

Dev Summit 2010, Image Services, and Silverlight

2010-04-05T01:30:00.000-07:00

About a week before the Dev Summit I was tasked with showcasing ArcGIS Server 10 image services in the plenary using Silverlight as the platform. The tricky part was limiting the temptation to “light it up” with Silverlight so as not to overshadow image services. And there was another problem - the last time I did anything related to image processing my cell phone was the size of an army field radio, Netscape was the browser du jour, and Enron was a hot stock buy. So I had to ramp up on image services quickly… which meant spending some quality time with the raster gurus at the factory (ESRI). Along the way, I gathered some valuable info and thought it might be interesting to relay a few pivotal gems on the use of image services and Silverlight in ArcGIS 10.

ArcGIS 10 supports a new geodatabase model, the mosaic dataset, which can easily and effectively manage small and large collections of imagery. They are created and maintained in ArcMap or ArcCatalog and you use a set of GP tools to manage. Mosaic datasets support imagery from multiple sensor platforms including QuickBird, IKONOS, and WorldView. Most importantly, there is no preprocessing required. Mosaicking, pansharpening, and ortho-rectification happen on the fly. Merely add the image to the mosaic dataset and it assimilates.
Mosaic datasets support time awareness. Image services are time aware if the underlying mosaic dataset has been time enabled. You enable time awareness in the mosaic dataset properties, not the layer properties (as with feature layers in a map service).
Sometimes rasters in a mosaic dataset will not display at full or larger extents. In this case, overviews must be generated. Overviews are similar to pyramid layers in that they allow for the quick display of very large collections of imagery. Overviews do this by resampling source data to generate "snapshot" images for larger map resolutions (larger extents). In most cases, you will want to generate overviews. However, this poses a problem when you want to display a set of rasters that occupy the same space in a time series. Overviews are generated for a specific map resolution (map units/pixel) ranges, not a specific instant or span of time. Therefore if you're viewing your image service at a map resolution that uses an overview, the overview will only show one instant of time regardless of the time extent you specify. In this situation, avoid generating overviews and increase the maximum cell size range factor to enable the display of primary rasters at larger map resolutions. To do this, use the Calculate Cell Size Ranges option from the mosaic dataset context menu in ArcCatalog/ArcMap or use the geoprocessing tool of the same name. Increase the range factor to increase the threshold at which overviews are used, if available.
9.3.1 Image Server functionality is now part of core ArcGIS Server 10.
All rasters, including mosaic datasets, can be published as image services. An individual image on disk is termed a "raster dataset" or "raster layer".

Image services include an enhanced export image operation and three new operations: query, identify, and download. Each are covered in detail below.

The export image operation has a new JPGPNG image format which lets the image service define the ideal output format. If the extent requested contains transparent portions, generate a png. If not, generate a jpg.

The export image operation uses mosaic rules to order, filter, and select tiles displayed in the mosaicked image. For example, if a service hosts a set of overlapping images you can define which ones are displayed over others based on an attribute such as date, or a point of interest. The REST API documentation provides minimal insight into parameter descriptions. The SOAP API documentation may provide more information (it definitely will in ArcGIS Server 10 final). Also note, the identify operation uses mosaic rules to order and filter rasters to modify results.

The export image operation provides rendering rules to dynamically change the display of raster data using a set of predefined raster functions. For example, you can apply a custom aspect, slope, color map or shaded relief on the fly. Only 8 raster functions are available with the ArcGIS Server 10 REST API. One item you may notice in a rendering rule is the variableName attribute. It must be set to the value defined in the REST API doc, either “DEM” or “Raster”. It represents a property defined in ArcObjects where values may include “Raster2” or a user defined name for a function chain. These other values are not exposed in the REST API.

The query operation enables clients to define an attribute and/or spatial query and return information about rasters in a dataset hosted by an image service. You can also use it to return raster footprints or a raster thumbnail.

The identify operation enables clients to return a pixel value for rasters hosted in an image service given a user-defined location. The location can be a point or polygon. If a polygon, the centroid of the polygon is used. The order of rasters during the operation can be defined by a mosaic rule. Unlike the identify operation on a map service, image services do not provide a tolerance since only one pixel value per raster can be returned. Do not let the pixel size parameter fool you; it defines a pixel resolution to use when identifying pixel values. This means you can identify on overviews available in the mosaic dataset, and return an interpolated value in an overview image. Interpolation occurs when the overview is generated by resampling source rasters.

The download operation basically enables clients to download individual rasters and metadata hosted by an image service. You can also clip imagery on the server before download and define the download image format. If you don't want users to download individual rasters, you can easily disable this capability of the service.

So what does this mean for the ArcGIS API for Microsoft Silverlight/WPF? To put it simply, the ArcGISImageServiceLayer has been updated to support working with ArcGIS Server 10 image services. A set of new classes will be included to support mosaic and rendering rules. The QueryTask can be used with the image service Query operation. The Identify operation will require a new task, and the Download operation may be supported in the near future. 2.0 public beta will be released in April and you'll get a chance to try out some of this new functionality.

Also, at some point in the near future a set of ArcGIS Server 10 image services will be hosted online by ESRI to demonstrate their functionality. They may or may not be officially announced. In any case, I’ll tweet when they’re up, running, and public (@rex_hansen).

ESRI's European User Conference 2009 - Asides and Slides

2009-12-06T03:01:00.000-08:00

In October 2009 HNIT-Baltic hosted ESRI's European Users Conference in Vilnius, Lithuania. I attended as an ambassador for ArcGIS Server, its Web APIs, and MapIt. This was my first foray into eastern Europe, and it turned out quite simply fantastic. Discussions with users and partners were spirited and exciting, and attendees in my sessions were engaged. The folks I met that work at HNIT-Baltic were exceptionally knowledgable and courteous, and simply ready for the next challenge.

The city itself was inviting, fresh, and personal. And I discovered a few things during my adventure: Lithuanians enjoy pizza with ketchup, because I was told it "adds flavor"; shopping malls look like factories on the outside, but they rival the best of So Cal on the inside, with better prices; and if you see a glass of water on a table at a party... it's probably vodka.

I've included a montage of a few photos I snapped with my trusty iPhone while in Vilnius (press "P" to pause).

If you attended the EUC in Vilnius or just want to view the conference materials, you can view and download the plenary and paper session materials from HNIT-Baltic.

To peruse the technical sessions I presented, I've made PDF versions of the session slides available below:
Building Mashups with ArcGIS Server
Building Great Web Map Applications
Whats New in ArcGIS Server 9.3.1

Using the ArcGIS Server REST API in .NET applications

2009-11-01T01:33:00.000-07:00

Over the past year I've handled a number of questions regarding the use the ArcGIS Server REST API in a .NET application. It can be done quite effectively, but isn't explicitly promoted by ESRI since the REST API is primarily designed to be consumed by pre-packaged Web clients like the ArcGIS JavaScript, Flex, and Silverlight/WPF APIs. However, using REST programmatically in a .NET application context without installing an ESRI product can make for some powerful and focused yet lightweight solutions. In addition, if you need to do something relatively simple you don't have to generate a load of client-side proxy classes like you do with SOAP.

I've included a simple code example below to highlight how easy it is to integrate ArcGIS Server services via REST in your .NET app. The example uses an ArcGIS Online geocode service to geocode an address and return a location.

Download the sample here.

There are a few items to consider when using ArcGIS services via REST in .NET:

You will construct the url that will execute an operation on the server. Inputs to the operation are included in the url as parameters with values. Add a reference to the System.Web.dll and use the HttpUtility class to encode parameter values in the url.
In most cases you will return results in JSON (JavaScript Object Notation) format. Add a reference to the System.Web.Extensions.dll (included with .NET 3.5, separate install for .NET 2.0) and use the JavaScriptSerializer class to deserialize JSON results into a dictionary you can use more effectively.

This example can easily be modified for use in a .NET CLR trigger for SQL Server to geocode address information upon entry into a table -or- in a batch process to geocode addresses in a database -or- show a location in a WinForm or WPF desktop client. The same idea can cross over into other ArcGIS Server services such as querying layers in a map or executing a geoprocessing service. In any case, you'll find the simplicity of the REST API refreshingly simple for those scenarios where you need to plug a little GIS into your app. One parting note, if you use ArcGIS Online services via REST, you'll need to follow the subscription model for the Web Mapping APIs. For example if you geocode more than 1000 addresses, you'll need to purchase a subscription.

"Price check on Bing Maps"

2009-09-24T00:14:00.000-07:00

Over the past couple months, I've chatted with a number of people interested in using Bing Maps services in their applications; map-centric and otherwise. If they had any questions or wanted to purchase a license, I always mentioned that they could contact Microsoft directly. This got me wondering, if someone did contact Microsoft with questions about purchasing a Bing Maps license, what would they experience? Simple enough to try – so I called their contact number listed on their licensing page.

I was directed to the voice mail of someone on the sales staff. His voice mail greeting provided his email address, so I decided to send him an email with a single question to get the ball rolling. Literally 7 minutes later I got a response… I was impressed. He needed more information, so I provided it and waited for a response. The following day I sent an email asking for an update and he called me within 5 minutes. Evidently he called my main work number and made his way to my desk… quick, resourceful… again I was impressed. After a lengthy conversation, a few interesting details about the licensing and cost of Bing Maps became apparent. Let’s start with the first question:

After the 90-day evaluation period, do I have to purchase a license to continue using Bing Maps imagery, geocode, and route services?

No, but the use case is very limited. You can continue to use Bing Maps staging services beyond the 90-day evaluation period if the application is:
1) available publicly, on the Internet, without restriction
2) completely non-commercial, meaning you make no money on its use, directly or indirectly

Note you still have to abide by the limits defined for the 90-day evaluation. So this pretty much covers soccer moms trying to organize team events on a map or allows your friendly fantasy football commissioner to show folks how to get to his house for the draft. Granted the Bing Maps imagery services will still have “Staging” plastered on each tile, so the map may not be very pleasing. Outside of this, all other scenarios require a license. This includes all internal applications (commercial or non-commercial, Web or desktop), all commercial applications, and all government applications. Logically this leads to the next question…

How much is a license to use Bing Maps?

Short answer…the standard package is $8000 for 1,000,000 transactions. This is the minimum to get started. The transactions sit a pool for use with all Bing services: imagery, geocode, routing, search, etc. So…

What is considered a transaction?

It depends on the service. The best place I’ve found to get detailed information on this is the transaction report description for Bing Maps. In general, use the following as a guide:

8 map tiles = 1 map transaction
1 geocode = 1 transaction
1 route = 1 transaction

In the end, I was both impressed and satisfied with the responsive Bing Maps customer service. The guy I spoke with was clear, concise and real; he knew the technical details and presented it well. I’m definitely more comfortable referring folks to Bing Maps now that I know first hand what they’ll experience. I still find it interesting that if you are using a licensed ArcGIS product or API you can purchase access to Bing Maps through ArcGIS Online for significantly less - $2500 for 1,000,000 transactions (see the price sheet). The only difference is ESRI customer service provides you with a Bing Maps account and you can purchase transactions in smaller blocks (100,000). In your application you still work with Bing services directly. We'll see what happens moving forward, but for now ESRI customers appear to have some options when buying into Bing Maps.

ArcIMS equivalency in ArcGIS Server?

2009-08-23T00:41:00.000-07:00

In May of 2000, ArcIMS 3.0 was released and over the next few years it took the Web GIS world by storm. There were few competitors, so the market was ripe. While it was considered complicated initially, most folks were able to buckle down and wrap their head around the technology - and build some rock solid solutions. HTML Viewer based applications, servlet connectors, and custom solutions using the ActiveX, ColdFusion or Java connector started popping up all over the Web. Turns out, ArcIMS was actually simple – all interaction was stateless and anything it could do on the server was dictated by ArcXML – so its boundaries were well defined. Plus it did its job very well. Dynamic mapping, feature streaming, queries, geocoding, extract... all were quick and efficient.

In the Web mapping world of late 2009, ESRI brings its wealth of GIS intelligence to a party that includes companies originally founded on search engines, operating systems, and business software. The market has expanded dramatically since ArcIMS was first released, and it continues to grow. ArcGIS Server now provides a comprehensive enterprise GIS solution for the Web with seemingly unlimited capabilities, especially when you factor in access to ArcObjects on the server. In general, it integrates with a diverse range of back-end data stores, utilizes an effective and robust authoring environment to build data and logic into a service, and it exposes numerous public Web service formats and protocols to support a wide range of consumers. In addition, a bevy of Web clients are available… server-side AJAX solutions using the ADFs, lightweight browser clients using the JavaScript API, and rich Web clients using Silverlight or Flex. With the sheer volume of features, functionality, frameworks and APIs in the ArcGIS Server suite, it remains unmatched as a complete enterprise GIS solution.

So with all this said, we know that ArcGIS Server can do so much more than ArcIMS in many areas, but are there still some things ArcIMS can do that ArcGIS Server can’t – or can’t do very easily? Yes. Let’s step through the ArcIMS service (virtual server) types as see:

Geocode? Yes, ArcGIS Server has it covered in the geocode service.

Route? Yes, ArcGIS Server has it covered in the network analyst extension on a map service.

Metadata? Yes. The GIS Portal Toolkit was added as an extension to 9.3. At 9.3.1 it was termed the ArcGIS Server Geoportal extension.

Extract? Yes, but it’s not as simple as ArcIMS. In ArcGIS Server you can emulate an extract virtual server with a custom tool (model) in a GP service. Granted GP can be very complex, but also very powerful... a good trade-off. Geodata services do provide an extract option, but the service was primary designed for the ArcGIS desktop client and functionality is limited to working with a geodatabase.

Feature? Yes, except it’s handled via a query operation on a feature layer in map service. The feature virtual server was primarily designed for the Java Viewer - the “rich” Web client when ArcIMS was first released. ArcGIS Server 9.4 will include a feature service with much greater functionality that that provided in ArcIMS.

ArcMap? Not quite. In general the ArcMap virtual server just exposes a subset of ArcGIS Server map service capabilities... so generating dynamic maps and querying layers is covered. One notable difference involves access to page layouts. In ArcIMS you can use GET_LAYOUT to generate printable output from a page layout in an mxd hosted by an ArcMap image service. In ArcGIS Server this capability is not available via a Web service protocol or format. Instead you must use ArcObjects (remote) and QI to IMapServerLayout from the map server object -or- use the Web ADF's PageLayout control which only works with local ArcGIS Server data sources. In either case interaction usually remains stateless.

Image (map) and query? No, but ArcGIS Server is crossing things off the list. On the server customization side, ArcGIS Server is unmatched. You can extend a server object in ArcGIS Server using .NET or Java –or- create and access ArcObjects remotely. In ArcIMS customizing a virtual server using C++ was rare and relatively difficult. There's also the issue of performance. Dynamic mapping in ArcIMS is notoriously fast. To match and exceed ArcIMS performance, ArcGIS Server provides the ability to pre-cache map tiles (9.2+) and to generate “fast” dynamic maps with optimized services (9.3.1+). One notable addition to ArcIMS 9.0 was the ability to reproject geometry using GET_PROJECT. This was covered in ArcGIS Server 9.3 with the release of the Geometry service.

With this out of the way, let’s focus on comparing the stateless capabilities of ArcIMS image\query and ArcGIS Server map services. Below I’ve listed six stateless capability categories that are popular in ArcIMS and still deficient at some level in ArcGIS Server:

Reorder layers (including graphics) – It’s possible in ArcGIS Server but requires a stateful change and you must work with fine-grained ArcObjects.
Change layer renderers – It’s possible in ArcGIS Server but requires a stateful change and you must work with fine-grained ArcObjects.
Enable use of variable transparency for symbols (applied to graphic elements, selection sets, etc.) – Right now symbol transparency can be either 0% or 100%.
Add dynamic layers – It’s possible in ArcGIS Server but requires a stateful change and you must work with fine-grained ArcObjects. If dynamic layers can be added, two additional capabilities will be necessary for equivalency:

Enable joins in dynamic layers
Query dynamic layers

Support related data in query results - All ArcGIS Server map service query operations support returning joined data. In 9.3 the identify operation returns related data (SOAP only), but other query operations do not. So, to retrieve related rows in an ArcGIS Server map service right now, you have to use fine-grained ArcObjects and traverse the relationship class collection on a feature layer. This will change in 9.4; query operations will return related data.
Fully support scale dependent rendering on a single layer - This is relatively easy to do with ArcXML in a map configuration file, but in ArcMap it requires VBA and the symbology is not readily accessible – namely the Toc swatches are empty. A better solution involves using a group layer in ArcMap. Essentially you add a reference to the same layer multiple times within a group and set the scale dependency and symbology for each layer. Unfortunately all the layers show up in the Toc (even those outside scale range) so the experience isn’t exactly like ArcIMS, which seems cleaner.

For the most part, ArcIMS equivalency can be found in ArcGIS Server. My interest in posting the exceptions is three-fold: to provide awareness of these issues, to mention resolutions pending in future versions, and to provide guidance on a current solutions or workarounds. If you’re aware of another ArcIMS equivalency issue in ArcGIS Server, I encourage you to add a comment below. Hope this was helpful.

ArcGIS Silverlight/WPF API and ESRI MapIt blogs up and running

2009-08-14T15:45:00.000-07:00

The ArcGIS API for Microsoft Silverlight/WPF and ESRI MapIt, two new products in the ESRI suite, were released during the 2009 Users Conference in early July. Now, each has it's own blog hosted by ESRI.

Silverlight/WPF Blog: http://blogs.esri.com/Dev/blogs/silverlightwpf/
ESRI MapIt Blog: http://blogs.esri.com/Dev/blogs/mapit/

Both are available their respective resource centers as well (Silverlight/WPF, MapIt).

I'm the lead product engineer for both products, so I've been busy working with a group of extremely talented developers at ESRI to build and get them out the door. The ESRI blogs should provide you with some great information about both products... good times ahead for sure.

Consuming ArcGIS Server SOAP services in Silverlight 2.0

2009-01-24T15:34:00.000-08:00

Microsoft's Silverlight platform offers yet another application environment in which to consume SOAP services. The Silverlight platform is truly tantalizing - the ability to blend\mash\mix services and data with a rich user experience in a web client is undeniably sexy and powerful. Unfortunately some of the backend plumbing still needs to be worked out. This includes Silverlight's ability to consume SOAP services, let alone ArcGIS Server's SOAP service stack. Sure, everyone's talking about REST and WCF, but W3C standard, contract based, explicitly typed interaction with SOAP services is still prevalent - just take a look at the recent release of the Virtual Earth Web Service SDK.

So how do you consume a SOAP service in a Silverlight application or class library? Once you install the Silverlight tools with Visual Studio (or Web Developer) 2008 sp1 you will be provided with a context menu item off the project in Solution Explorer. The item is named "Add Service Reference" and opens a dialog box for a developer to define the endpoint to a service (WCF,SOAP) and a namespace for the native client types that will be generated.

This capability is synonymous with the "Add Web Reference" capability for SOAP services in standard .NET projects. A single url to a WSDL is used to construct a SOAP proxy and a set of value object types. This provides a nice interface for one-off services, but if you have a set of services that share value object types and you want to generate a single library with a single namespace, you need to use a command line tool. This is possible with the NET 2.0 and 3.5 SDKs which include the web service utilities wsdl.exe and svcutil.exe, respectively. Note, wsdl.exe output is supported in .NET 2.0 - 3.5. Unfortunately Silverlight does not provide the same or similar utility. Before Silverlight 2.0 final was released, a utility named slwsdl.exe was available for this purpose. Unfortunately it was removed from the 2.0 final release for reasons unknown. In addition, the output from wsdl.exe and svcutil.exe will not work with Silverlight applications. Both wsdl.exe and svcutil.exe generate references to types that are not available in the Silverlight platform. Problematic type references generated in svcutil.exe output can be removed so reference code can actually be built as a Silverlight class library. Unfortunately the proxy class includes both synchronous and Begin\End asynchronous methods. Neither pattern is supported in a Silverlight application. In fact, the Silverlight "Add Service Reference" tool only generates Async\Completed asynchronous methods, which is the "promoted" pattern for event-based asynchronous programming. Apparently there is no way to trigger svcutil.exe to only generate the appropriate asynchronous methods for Silverlight and synchronous methods are not officially supported in Silverlight, apparently due to cross-browser support issues.

So with all this in mind, how can you generate a single library that contains all the proxy classes and value object types for all ArcGIS Server service types? It’s hokey, but you need to manually merge all WSDLs into a single WSDL – being careful to place the element types in the same locations (e.g. element vs. operation, etc.). You only need to do this once since the library you generate will be distributable thus reusable for all Silverlight applications\libraries. In fact, the sample included with this post includes the reference class file generated from this process, so you don’t need to mess around with the ArcGIS Server WSDLs – just download the sample and compile the ArcGIS_SOAP_Silverlight project. Or just use the precompiled ArcGIS_SOAP_Silverlight.dll in the bin folder.

The sample is available here.

The sample also contains a Silverlight application that illustrates a simple use case for consuming an ArcGIS Server dynamic map service and navigating the map. Left mouse click on the map zooms in, Shift+left mouse click zooms out. Note you’ll need Visual Studio 2008 sp1 and the Silverlight tools to load and build the projects in the solution (see http://silverlight.net/getstarted for more info). You can give a test run below. It's designed to be instructive, so it's pretty simple.

One additional note, the Web site that hosts the ArcGIS Server services must have a clientaccesspolicy.xml(Silverlight) or crossdomain.xml(Flex) in place to support cross domain\site requests. Silverlight will work with either. If using crossdomain.xml the following entry will enable SOAP interaction for all consumers:

<allow-http-request-headers-from headers="*" domain="*">

ArcGIS Web ADF, JavaScript, and REST

2008-11-29T23:59:00.000-08:00

Currently the Web ADF does not include an ArcGIS Server data source that supports consuming ArcGIS Server services via REST. Instead, the pre-packaged ArcGIS Server Internet data source uses the SOAP API to consume ArcGIS Server services. From a platform perspective, this makes sense because the Web ADF is founded in a rich, server-side ASP.NET development environment where SOAP provides some clear benefits. Let's look at this topic in more detail...

REST is a lightweight, non-standard format which can be used to request data over HTTP. REST is especially attractive to browser developers because it can be leveraged with JavaScript, JSON, and HTML to create an efficient, pure client solution - namely one that uses only browser logic and can initiate cross-domain requests for data. In general, requests contain a manually concatenated string of argument-value pairs and responses consist of standard content types, such as JSON (text) and images. While the benefits of using REST in a browser context with JavaScript are clear, other application environments present some hurdles. Since REST is not founded on a public standard, a developer needs documentation to learn how to interact with a REST service. This may suffice when working with client script, but developers working in an IDE with object-oriented languages like C#, VB.NET and Java often demand more. Why parse strings when you can work with explicit service types, discover parameters using intellisense, and utilize a proxy to manage requests\responses? This is where SOAP comes into the picture. SOAP is a standard W3C procotol which defines the XML message format for requests to and responses from a SOAP service. WSDL, another W3C standard language, provides the contract for how a consumer can interact with a SOAP service. While a SOAP message is an XML formatted string, a developer does not need to construct and parse it manually. Since SOAP is based on standard protocols and languages, toolkits for specific development environments are available to automatically construct native types. More specifically, the WSDL is used to construct a proxy class and a set of supporting types for use in an specific object-oriented environment. The proxy class exposes a set of operations (methods) which initiate Web requests. The supporting types are used to define inputs as value objects (termed 'value objects' because they only store values), which the proxy uses to construct a SOAP message. The response from a service can be deserialized by the proxy into one or more value objects. The benefit here is clear; a developer is able to use complex service-specific types native to their development environment to interact with a service.

The key difference between the use of REST and SOAP lies in the WSDL. The WSDL provides a central, language agnostic means for defining how to interact with a SOAP service. Different development environments can use the WSDL to construct the same set of types on-the-fly using a SOAP toolkit. REST does not define a compliment to the WSDL. As a result, documentation must be used to determine request and response content. Technically, you can manually construct a proxy and set of types to interact with a REST service, but without a central definition or standard (provided by a WSDL) the API structure will likely differ between developers.

So where does it make sense to leverage REST services in the Web ADF? REST services should be utilized in the browser, thus integrated with the Web ADF using the ADF JavaScript library. In this capacity, REST services may be used to enhance Web ADF application behavior and performance. One such area involves simply adding non-cached map services using pure client logic, something the Web ADF does not provide out-of-the-box. Currently non-cached map services use the AdfMapHandler class in ADF JavaScript to generate dynamic map images. This class requires Web-tier components; namely a map resource associated with a MapResourceManager and Map server control. It uses the ADF MapHandler to generate dynamic maps. The same applies for the Toc; it still relies heavily on Web-tier manipulation. You can, however, extend the ADF JavaScript DynamicLayer class to work with a non-cached service via REST.

The sample is available here.

There are a few caveats:

1) The layer (i.e. resource) is only available on the client; server components do not know about it. As a result, any requests to the server that work with map resources will not include this layer. In addition, most ADF controls require modification on the server to change their rendered content. This includes the Toc, so if you want the pure client layer to show up in an ADF Toc, you’ll need to customize Toc content using code on the server.

2) Using services secured with token-based authentication requires a proxy. Here’s a link to step on how to set this up: http://resources.esri.com/help/9.3/arcgisserver/apis/javascript/arcgis/help/jshelp_start.htm#jshelp/ags_proxy.htm. Using a proxy is a popular solution for working around cross-site scripting restrictions in a browser. Abe Fettig has researched a number of these solutions and presented them in a blog post. Note, you can use a proxy even if authentication is not required.

In the sample I provided, if a proxy (via the proxyUrl property) is defined when creating a new DynamicRestLayer, the proxy will be used. In this case, a POST request is always generated to guarantee that all the input arguments are passed to the remote site. The response is JSON and contains the properties of the generated map image, including the url and extent. If no proxy is specified for the DynamicRestLayer, a simple GET request for a map image is made via a dynamic image tag. The image tag source is not subject to cross-site scripting restrictions in the browser, so we don’t need a proxy. Both IE7 and FireFox 2+ work great with the no proxy solution. Unfortunately IE6 is somewhat problematic (e.g. triggers erroneous requests). If you need to support IE6, you may want to go the proxy route.

This brings up an interesting point - the ADF JavaScript Map component is designed to render and blend maps using dynamic image tags. The image tag source is not subject to cross-site scripting restrictions. Since a REST map service can return the raw map image, the sample included with this post merely builds on REST capabilities and ADF JavaScript architecture. Another option involves using dynamic script tags whose source references a REST endpoint that generates JSON. The data returned from the REST service is evaluated as a JavaScript object when the tag is loaded or inserted in the page. This could also work with ADF JavaScript, but would require more implementation code. Interestingly, dynamic script tags are leveraged by the ArcGIS JavaScript API.

Note, the sample also shows how to use REST to access tiles in a cached map service with a pure client layer. It may be of interest since the url is standard across sites (e.g. you don’t need to know the virtual cache directory).

Web ADF JavaScript: How to determine event argument contents

2008-11-10T02:06:00.000-08:00

As many Web ADF developers already know, the 9.3 release contains a new public JavaScript library built on the Microsoft AJAX library. A host of ADF client controls and components (e.g. Map, Toolbar, etc.) maintain event handlers which enable ADF developers to listen for and interact with actions solely on the client (browser). Unfortunately the ADF JavaScript library reference does not contain the event argument types for event handlers. While this will be rectified at some point in the future, there is another option. You can determine the event argument types by looking in ADF JavaScript debug files installed in two locations:

<IIS Root>\aspnet_client\ESRI\WebADF\JavaScript
<ArcGIS Install>\DotNet\VirtualRootDir\aspnet_client\ESRI\WebADF\JavaScript

Search for references to this._raiseEvent in all debug JavaScript (*.debug.js) files. The first parameter to the _raiseEvent function is the type of event (e.g. 'click'). Any subsequent parameter is included as an event argument to any handler. You can look at the code to determine what the event argument contains.

For example, in the ESRI.ADF.UI.Map.debug.js file you’ll notice the click event is raised in the _onMouseUp function (shortened for the example):

_onMouseUp : function(evt) {
  var e = this._makeMouseEventRelativeToMap(evt,true);
  . . .
  this._raiseEvent('click',e);

The browser event is modified in the _makeMouseEventRelativeToMap function to set its coordinate property to a new ESRI.ADF.Geometries.Point as the location of the user click on the map client control. In this case, the _raiseEvent function for 'click' indicates that an event argument is included, is modified, and the type is not an array. In other cases, no event argument is passed (e.g. map's extentChanging event) or the event argument contains an array with many objects (e.g. map’s extentChanged event).

The first parameter to an event handler is the sender, which is the client component on which the event was raised. Any subsequent argument, if defined, matches the event argument in the call to _raiseEvent.

ArcGIS Server: Expose a custom server object extension via SOAP

2008-11-02T01:10:00.000-07:00

Extending ArcGIS Server using ArcObjects can take two forms: a utility COM object or a server object extension. A utility COM object is simply a COM component created and managed explicitly in server context. A server object extension (SOE) is coupled with a server object and thus can share initialization, resource utilization, and disposal. Use of a COM utility has been possible since the first version of ArcGIS Server (9.0). Custom SOEs were introduced for ArcGIS Server .NET at version 9.2. ArcGIS Server Java does not offer an SOE solution at the moment, but it is likely forthcoming. The reasons for extending ArcGIS Server and differences between both techniques are available elsewhere so I'll skip that discussion here. I want to discuss exposing ArcObjects logic in a custom server object extension via a prepackaged ArcGIS Server Web service handler. Two major protocols are supported, SOAP and REST. Let's dive in...

The ArcGIS Server SOAP API is implemented at the server object level. This means that SOAP messages are processed directly by a server object or extension (SOE). The ArcGIS Server SOAP Web service handler directs SOAP messages to a server object for you. The ArcGIS Server REST API is not implemented at the server object level. The ArcGIS Server REST Web service handler processes restful requests and (in general) uses the ArcGIS Server SOAP API to communicate with ArcGIS Server. How does this relate to exposing a custom SOE as a Web service? The SOAP Web service handler contains logic to discover custom SOEs with a SOAP interface (see sample and discussion below). The REST Web service handler does not contain the logic to discover custom SOEs at this time. Consequently, to expose a custom SOE as a REST service you’ll need to create your own Web service to handle restful requests and work with the SOE. Unfortunately this means you'll need to work with ArcObjects remotely in the Web service logic, which will require an ArcGIS Server license in the app-tier... a less than optimal solution. The folks at ESRI who work on the REST handler may resolve this in the future.

I've included a sample that illustrates how to expose a custom SOE via the ArcGIS Server SOAP Web service handler. It builds on the current Server SDK sample which shows how to create a simple SOE.

I’ve modified this sample to illustrate how to support exposing a custom SOE using an ArcGIS Server Web service endpoint. The sample is merely instructive, so it just provides the basics of custom SOE development. To expose a WSDL for the custom SOE and make it available via an ArcGIS Server Web service endpoint, create a WSDL and put it in the \XmlSchema folder on the machine where the SOM is running. The name of the registered server object extension and the name of the *.wsdl file must be the same. To process SOAP requests, implement the IRequestHandler and IRequestHandler2 interfaces in the custom SOE class. In this example, the implementation code parses the incoming SOAP request and generates a raw SOAP response string. The response should match the WSDL definition.

Download the sample here.

Everything is pretty raw and simple at the moment, but it works. I’m sure there’s a better way to create the WSDL and generate SOAP responses, but that’ll take some additional research. Once the custom SOE is deployed and enabled on a map service, you can use the following url to get the WSDL: http://localhost/arcgis/services/<service name>/MapServer/SimpleSOE_CSharp?wsdl

Inaugural Post

2008-10-29T14:44:00.000-07:00

I thought it might be time to start posting a few diatribes on technologies I encounter and wrestle with on a regular basis. Hopefully this blog will introduce some interesting topics related to the mesh of ESRI server technologies, Microsoft products, platforms and components, and just about any other Web standard, protocol, format, API, etc.

Hope it works :)