Another Cartography Blog

http://support.esri.com/en/knowledgebase/techarticles/detail/40913

I always forget how to deal with nulls - must I treat it as a string or use the IS statement, then how do I need to reference the null value?  Is it case sensitive? I found this post in the technical reports on the ESRI support website.  It is nothing new, but a nice quick reference on how to check for null values in my attribute table.  I'm using 10.1 and didn't have to specify to use a Arc 9.3 Python parser (but then again, that may be the default...)  Anyway, it is a super simple definition to put into a Python Codeblock:

def IsMyValueNull(field): if field is None: return UpdatedValue else: return field

The past few days, I’ve had my nose buried in the new Animal Crossing installment on the Nintendo 3DS.  I’ve been a fan of the game since it was released on the GameCube and have always loved the task of creating a “perfect” town.  This is a game feature that calculates the synergy between the distribution of trees and flowers, and now public works.  It is always a difficult task; I find it much easier with a detailed map of my town.  This allows me to easily see the spatial distribution of trees, identify areas that are too dense or too sparse.  In the past, I’ve always sketched my town onto graph paper because it suits the gridded structure of the game.  However, sketching the map makes it difficult to update as I modify my town’s layout to create the ‘perfect’ rating.  This time, I chose to create the map in excel, a rather strange venue for cartographic production.  It was perfect for automatically calculating the number of trees in my town and updating it as villagers move in and out.  Though Excel still caused some issues since I wasn’t able to incorporate any angles into the map, and I’ve experienced some strange “null” spaces in the map where the cliffs and rivers bend and curve.

I just tried to change the color of a layer in my ArcMap session and my color selector palette has gotten corrupt; all of the default color swatches had disappeared and were replaced with many "odd" colors. After a moment’s panic, it was rather clear where all of the bogus colors had come from – my recent overuse of the eye dropper tool. The eye dropper tool in ArcMap can be a very useful feature for many reasons. I’ve been using it to test the apparent RGB colors on some transparent layers (attempting to avoid the known issue of flattening PDF exports when transparency is applied while raster layers are also in the data frame).

http://www.esri.com/library/brochures/pdfs/arcgis-desktop-tips.pdf

The other day I was given a map package file (that included data) and told that the dataset was too big and needed to be clipped before it was released. The data to be distributed only supposed to cover a few square miles in a square extent frame, yet the datasets originally extend far beyond those limits. Although some masking parameters went into the input of the Data Management “Package Map” tool, it doesn’t seem to clip the data as expected. What appears to happen is that ArcGIS performs a selection on anything inside the clipping parameters and extracts that selection for the package. This makes perfect sense for some political information (municipality boundaries) where I don’t want to force data to be the extent of my project. However, this leaves me with dangling linear features! The ArcMap 10.0 project consisted of about 35 layers – no way that I would want to process this data myself, especially if there is more than one map package file. So, I started to dig into ArcPy’s Mapping module to process each layer in the document. However, in the [too many] hours I spent trying to figure strange problems out, I thought it be worth the while to pass on what I’ve learned. Lesson 1: Scheme Locks It is possible to unpack the map package in ArcPy, but doing so may cause some schema locks on the data. Later, I realized that there is an “Extract Package” tool. I’m not sure how different this is to my method or how many similar problems might be experienced with using it. For all I know, it would have likely removed all of my headaches! The way in which I unpacked the map was as follows:

import arcpy, mapping inFile = “C:\\ … myLocalSaveSpot… .mpk” MPK_file = inFile MPK = arcpy.mapping.MapDocument(MPK_file) MXD = MPK.filePath del MPK myMXD = arcpy.mapping.MapDocument(MXD)

It is crucial that you delete the MPK variable. While this is still in memory, it puts a scheme lock over the data that it extracts. Thus, if you try to process any of the data, they will error out. Lesson 2: Processing and updating data sources The next step I needed to do (the important one) was to update the data by clipping it. It is again important to note my purpose here – I wanted to replace the data that existed in my map package. This is a bit “off” because the purpose of the map package is to take a snapshot of your data and either archive or share it. Indeed, I need to share a snapshot of my data, but it needs to be clean! The second step here is to update the data. I don’t want to remove the layer from my ArcMap documents and re-add the new one. This would destroy the symbology that is set. So, what needs to happen is that the original data must be deleted and the ArcMap layer must be resourced. I had a heck of a time using the arcpy.mapping.replaceDataSource to work (I think my problem was trying to set the incorrect workspace type…), so I came up with another method. I clipped the data and saved it into the delightful memory workspace. I then deleted the original clip layer and replaced it with the clipped layer that was stored in memory. I found that this strategy was efficient because I wasn’t having to delete, save, rename, delete, etc. many times. The strategy I used was below :

df = arcpy.mapping.ListDataFrames(doc) lyrs = arcpy.mapping.ListLayers(df[1]) #I have several data frames in the document featLyrs = []#Create an empty list to store all feature class layers for lyr in lyrs: if lyr.isFeatureLayer(): featLyrs.append(lyr) #Need to extract only feature classes. Raster datasets will use a different clipping tool clpLyr = arcpy.mapping.ListLayers(df[1])[-1]#my clip extent is the last layer in the data frame for clp in featLyrs: if (int(str(arcpy.GetCount_management(clp)))>0: #Don’t want to clip an empty feature class savePath = clp.dataSource arcpy.Clip_analysis(savePath, clpLyr, “in_memory\\temp”) arcpy.Delete_management(clp.dataSource) arcpy.CopyFeatures_management(“in_memory\\temp”, savePath)#saves the in_memory fc to the exact location and name as the original layer arcpy.Delete_management(“in_memory\\temp”)

It is also possible to parse out raster or group layers if you need to. The call lyr.isRasterLayer() will return a true if, well, the dataset is a raster. Likewise, you can omit examining other things like group layers by the call lyr.isGroupLayer(). This fits well into the code while looping through data frames and the layers within. Lesson 3: Repackaging the Map Package Of course, I cannot change the fact that Map Packages are intended to be a snapshot of my data. This means that the data that I process will only be temporary and not be included in the original map package file if I share it. Unfortunately, the map needs to be repackaged. But what to do about that original, messy one? The last part of this code shows how to package the MXD into the original MPK file (kinda). Beware that if your dataset is large, this will still take some time (but if you had already created an MPK file, I’m sure you are well aware!).

MPK_revise = MPK_file[:-4]+"_revise.mpk" arcpy.PackageMap_management(doc.filePath, MPK_revise) os.remove(MPK_file) os.rename(MPK_revise, MPK_file)

Mount Meeker and Longs Peak from Longmont, CO

Recently, I came across a problem in which I needed to create a “unique” random number for each row in my attribute table. The situation was that I have a feature class which had thousands of features. For testing purposes, I wanted to randomly identify only 100 to sample from. There does appear to be function in the old Hawths Tools that will create a random selection. However, due to work computer restrictions, I’m not sure if I would be able to use the tools or easily implement them into existing workflows. Are Hawths Tools still compatible with the new versions of ArcGIS? Defining random numbers in ArcGIS is a frequent necessity. Luckily, this feature is becoming easier to implement as Arc evolves, especially in the ArcPy syntax. However, many aren’t familiar with using ArcPy to manipulate shapefiles or perform complex attribute calculations (which is a shame, you should learn). The field calculator is still a crucial tool for performing similarity complex attribute calculations. Sadly, using a basic random number generator won’t allow me to efficiently identify which features to examine. For example, in Code Block 2, random float numbers are generated between 0 and 1; it is likely that each number would be unique. But this is clumsy. In order to identify 100 random values, I would need to then perform a selection on the field from 0.0 through (e.g.) 0.334. Plus, the ending query number would likely be different every time I re-ran the random number generator. Alternatively, I could sort the field from low to high and then create a new field to and identify an integer order using an automatic incrementer (Code Block 1). However, now I have 2 fields just to get the random order. Code Block 3 is another way to do this, yet probably has more drawbacks in this situation. While it is nice to define a range of numbers (e.g. 1-100) to select from, it again is clumsy because I cannot directly control how many times “1” appears. This method could theoretically calculate “1” 500 times, leaving me in a similar situation from where I began. Finally, I was able to find a Python function that will generate a randomly ordered list of numbers! From this, I was then able to incrementally assign a number of the list to each row, shown in Code Block 4. This allowed me to maintain an integer number to identify which features were to be sampled. It also allowed me to easily re-run the random number generator over and over without fear and enabled me to efficiently change the number of features to be sampled (from 100 to 10 to 1000, etc.). Below are a few useful functions that can be copied and pasted into ArcGIS’s Field Calculator. These code blocks are to be pasted into the expanded “Code Block” section of the field calculator. The parser must also be changed to PYTHON. The field calculator text box then only needs to call the function (i.e. AutoIncrement(), as in code block 1). Note that for these functions, I’m using Arc 10.1. I’m not sure if/what changes should be done for any other versions.

Code Block 1: Automatic Incrementing

This code will assign an incremented number to the field. If a specific order is desired, the attribute table should be sorted first. This will number your rows from 1 to n.

i = 0 def AutoIncrement(): global i increment = 1 i += increment return i

Code Block 2: Get Random Number from 0 to 1

This simply generates a random floating value from 0 to 1.

import random def Rand(): return random.random()

Code Block 3: Get Random Number within a given range

This will generate a random number within a given range. The example has the range from 0 to 10. Note that this range is not inclusive and will only return values from 0 to 9. It can be updated to any range (I’m not sure if/what restrictions are in place).

import random def Rand(): return random.randrange(0,10)

Code Block 4: Get Randomly Ordered Number

This function will calculate a unique, random number. It is done by randomly sorting a list of values then assigning each value in the list to a row. It is EXTREMELY important that the code be updated to match your unique situation. In my case, I had 5000 records, so I needed a list of 5000 numbers. The two instances of 5000 below must be updated to reflect the total number of records in your table. Otherwise, the function will create out-of-range values or error out if the list ends prematurely. If this gets implemented into a Python script, the arcpy.GetCount() function can easily be substituted to automatically count the number of records.

import random randVals = random.sample(range(5000), 5000) i = 0 def Rand(): global randVals global i val = randVals[i]+1 i += 1 return val

Sand Dunes National Park

Over the past few semesters in which I have instructed an Introductory Cartography Lab, I’ve just about seen it all.  However, one of the most mysterious phenomena occurs randomly and without warning, or so it might appear.  This comes in the form of green lines that stretch from the map into oblivion.  There is a large “conspiracy” about what causes these lines and possible solutions (as documented below in two relevant ESRI Forum Posts).  However, I’ve experienced this problem coming yet another source (not saying any of the below are incorrect) and offer several ways in which to resolve or help prevent the green lines in the future.  So, coming from a background of watching about 200 introductory cartography students (with NO prior Arc experience), here are my commentaries on the following “solutions”:

In a post from 2005 (http://forums.esri.com/Thread.asp?c=93&f=1731&t=154572):

 The way CAD data is read into ArcGIS

 Problems during the exporting process

 Limitations of Windows graphics

I don't have much experience with CAD, so I cannot speak too much to this point.  The problems occurring during the exporting process seems unlikely simply because these green lines can be seen in Arc's Layout View prior to exporting.  It seems probable that it could be related to graphics limitations, but this could be an extremely complex scenario, if even possible to resolve.  The graphics limitations may also be linked to the "Bad Geometries" of data.

 In a post from 2013 (http://forums.arcgis.com/threads/65285-green-mystery-lines-in-layout-view):

1.  Copy and Paste entire data frame into a new Map Document

Next to starting from scratch with making a map, this seems like the obvious be-all-end-all solution.  But this may be slightly undesirable for several reasons and another solution may be better.

2.  Clearing Guides

While all of our Cart students have been shown how to use guides, this practice often goes above and beyond anyone just starting ArcGIS and has no background in any Design software such as the Adobe Creative Suite where gridlines are readily used.  This also seems unlikely because the gridlines are simple snapping lines, so I doubt this is causing any trouble.

3.  People suggest the “Repair Geometry” tool in the Data Management Toolbox

Again, this seems to be a plausible solution when dealing with bad geometries.  I’ve never used this too before, but looking at the Help Files, it seems like it might be good practice to use this too on just about any map document to ensure everything is in good standings.  It doesn’t seem like it does anything besides notify the user if there is any problem.

  What seems to be lacking:

As part of the cartography lab curriculum, we overview legend manipulation in two ways.  The first uses the Legend Properties and the second involves fine-tuning by Converting to Graphics.  Beginner students find the converting to graphics the easiest to deal with and tend to apply this just about everywhere they can.  I have found that this can be a major source of problems, especially if multiple things are grouped together and can result in the dreaded green lines.  So, from what I’ve seen, this problem appears to also arise from corrupt elements in the ArcMap Document.  I’ve determined (sometimes) that if the green lines form two sides of a triangle, they frequently are pointing to the corrupt feature.  This makes life easy because I can simply delete the corrupt element and re-create it.

 Let this be a note that this problem isn’t applicable to only newbies.  More experienced Arc users may also like to experiment with ArcMap Layouts.  Copy and Pasting, Expanding elements (converting to graphics), and other heavy modifications between Data View and Layout View are prone to corruption of Map Elements.  If a user is rapidly experimenting with new things, it is easy to forget at what point an element went corrupt and can be difficult to backtrack to eliminate the problem.

 Strategies to prevent Green Lines from corrupt Map Elements

Most importantly, VERSION YOUR WORK!  If something happens, don’t let it ruin the hours you’ve spent creating it.

Avoid grouping dynamic elements (data frames, scale bars, legends, etc.)

Only convert to graphics when necessary (i.e. don’t do it on scale elements willy-nilly)

Avoid readily moving figures and elements from data view to layout view

Don’t convert features to graphics in Data View and try to copy and paste a feature (i.e. stream segment or annotated road shield) to Layout View.  More often than not, you won’t be happy with what happens anyway; everything is scale dependent.

If you see the green lines, go through and delete one map element at a time (don’t forget to “Refresh” the layout view to redraw the green lines) and undo the delete.  If you see the green lines disappear, you may have found the piece of the document that went bad.  Recreate that one piece and continue working.

In the past, I have never had many issues with extracting X-Y data from excel files and using it in ArcMap.  However, I’ve recently been having difficulty properly exporting GNIS points from an Excel file (pipe delineated text file saved in a .xlsx worksheet) to a shapefile.  I’ve made sure to set the Lat/Long fields in Excel to Numeric Data types and set the geographic projection in ArcMap to WGS 1984, the data’s native coordinate system, as I display the XY Data.  The data appears to export just fine, but when brought into the same ArcMap session, the extent acts strangely when zooming in on in.  An error finally occurs when adding the data to a different ArcMap session – The error reads:

“Warning, inconsistent extent!

One or more of the added layers has an extent that is not consistent with the associated spatial reference information.

Re-projecting the data in such a layer may lead to unexpected behavior.”

Well, I think I have more problems than simply being concerned about projection transformations!!  After much searching on the web, it appears that yet again, Excel is partially to blame (and maybe some user error.  OK, mostly user error).  Because of unexpected characters (spaces, in my case), the data was unable to save properly.  My solution was to perform two search and replaces in the excel spreadsheet to format the data and remove these “unexpected” characters.

The first was to replace any spaces at the beginning of a cell; the second was to replace any spaces at the end of a cell with nothing (i.e. delete the space).  I achieved this by using the wildcard (*) function in the Find/Replace dialog.  To delete the spaces at the beginning, the Find What field should be filled in as *[space]; a [space]* will delete the spaces at the end of a cell.  This function will only remove one space for each time the Find and Replace is run.  It might have to be run multiple times in order to eliminate all the problematic characters.

For formality sake, I also ensured that my field names had an appropriate length of 10 characters.  Although, any field names longer will be truncated, it is better to be safe than sorry.

 ----

For basic information on adding X-Y data to ArcMap 10.1, check out the ESRI Help file:

http://resources.arcgis.com/en/help/main/10.1/index.html#//00s50000001z000000

In 1957, Arthur Robinson identified a problem with terrain mapping: it is difficult to represent terrain, an 3-dimensional feature on a flat map.  To overcome this difficult, he created a method called Inclined Plane Contours.  The method involves taking an existing contour map and sticking an imaginary plane through the map at a given angle.  by connecting the existing contours together using this plane, relatively horizontal lines can be created that depict the local relief.  The benefit to this is that the map represents a 3-D surface while being planimatrically accurate!  Above is Robinson's final product and below is my replication of the method (the high peaks on the right of the map are Mount Meeker and Long's Peak).  Because I did my replication in an automated environment, it lacks many of the sketch-like characteristics of the above map.

This map was published in Richard Edes Harrison's atlas, "Look at the World: The Fortune Atlas for World Strategy", published in 1945.  Just after the end of World War II, cartography for military purposes and propaganda was everywhere.  Harrison's maps take advantage of oblique view of the earth, allowing the reader to see the horizon, and greatly exaggerating vertical relief.  This map, Japan from Alaska is used to show the "great and terrible" threat Japan is and how close it is to the continental United States.  The awkward mapping angle and exaggerated relief create a feeling of great power and superiority Japan can have over the United States and how much of a threat this global power is.