Liz recently needed some shapefiles from an ArcGIS online map. Checking out the linked page, it’s immediately clear that there’s a lot of data, and no obvious way to get it from a download or share link anywhere on the app page. The desired solution is anything but taking a screenshot and tracing it in ImageJ, since that’s an absolute last resort.
In this post, I’ll walk through how I managed to get those shapefiles downloaded, and hopefully provide some easy tips to do the same for other ArcGIS online maps.
The power of the web inspector
This is fundamentally a web scraping task, and I’ll start with opening the web developer tools in Firefox, by right-clicking a promising bit on the page (the map itself) and selecting “Inspect”. Looking through the HTML tree in the web inspector panel that pops up, I can see that while the shapefiles do appear to exist locally, these are parsed into a gnarly embedded SVG object. This could be used to reconstruct the shapefile, but it seems like a big pain that I don’t want to deal with, so I move on from this avenue.
data with a query payload of
f=json. Inspecting that response object leads me to another API endpoint that appears to be what I want!
ESRI API endpoints
I’m actually fairly familiar with ESRI’s REST APIs, and I know that I can actually navigate to the API endpoint and it’ll provide a fairly good description of its data. I can also interactively query it in the browser, without having to muck about with cURL in Terminal or anything like that. ESRI is quite humane in this respect, but again, there doesn’t seem to be an easy way to download the full shapefile directly from this endpoint, and I don’t feel quite up to the task of writing out a shapefile by copying and pasting a bunch of stuff.
% pipx install esridump installed package esridump 1.11.0, installed using Python 3.10.8 These apps are now globally available - esri2geojson done! ✨ 🌟 ✨ % esri2geojson “https://services.arcgis.com/8df8p0NlLFEShl0r/ArcGIS/rest/services/FHA_Grades/FeatureServer/0” fha.geojson 2022-11-03 23:42:54,990 - cli.esridump - INFO - Built 1 requests using resultOffset method
Now to fire up R and see that everything looks right by plotting it.
> library(sf) Linking to GEOS 3.10.2, GDAL 3.4.2, PROJ 8.2.1; sf_use_s2() is TRUE > library(ggplot2) > xx <- read_sf(“fha.geojson”) > xx Simple feature collection with 74 features and 4 fields Geometry type: POLYGON Dimension: XY Bounding box: xmin: -77.188 ymin: 38.79005 xmax: -76.8772 ymax: 39.0666 Geodetic CRS: WGS 84 # A tibble: 74 × 5 FID Grade Shape__Area Shape__Length geometry <int> <chr> <dbl> <dbl> <POLYGON [°]> 1 1 E5 2268576. 5849. ((-76.90432 38.85715, -76.9018 … 2 2 G7 13563378. 19322. ((-76.93371 38.87391, -76.90942… 3 3 H2 7772476. 12002. ((-76.88671 38.90218, -76.89007… 4 4 H1 12964128. 20269. ((-76.90942 38.89269, -76.93095… 5 5 G1 6516531. 19844. ((-76.93428 38.88311, -76.93574… 6 6 C4 7199183. 16914. ((-76.93371 38.87391, -76.96229… 7 7 H2 7328078. 14489. ((-76.96229 38.85169, -76.97798… 8 8 E2 9790479. 21957. ((-76.98859 38.8399, -76.9885 3… 9 9 F2 5253684. 15352. ((-76.99618 38.85609, -77.00305… 10 10 H1 1810343. 8218. ((-76.97203 38.89815, -76.9833 … # … with 64 more rows ℹ Use print(n = ...) to see more rows > ggplot(xx) + geom_sf(aes(fill = Grade)) + theme_minimal()
It looks fantastic, and is ready for further data analysis now!
A more direct route
Based on the README example, I learned that the top-level object is called
app, and that layers can be obtained through a method on the
_layers key inside this object, which seems to have the relevant data that I’m interested in.
app.map._layers.FHA_Grades_4159.url // "https://services.arcgis.com/8df8p0NlLFEShl0r/arcgis/rest/services/FHA_Grades/FeatureServer/0"