6 Steps to Making Beautiful Maps

 


About FME
|   December 18, 2017   |   By Tiana Warner

Cartography by @redgeographics using FME

The best maps are those that embrace both science and art, making them both informative and eye-pleasing, both accurate and expressive. To make this happen, cartography has to include both automation and creative design— a process of gathering, filtering, manipulating, and styling data.

The challenge is achieving all of this in a smooth workflow, one that’s repeatable and easy to expand upon. Our developers have considered the most painful challenges at each step, like blending different sources, preparing and cleaning data, and automation (keeping the map up to date with a repeatable workflow), and created transformers to help ease the process. Let’s look at the steps involved in the cartography workflow and which transformers are the most useful.

1. Integration

First, pull data from whatever sources you have access to: GIS, CAD models, DEMs, etc. Consider what will make the information as rich as possible without interfering with the map’s clarity.

Examples of possible data sources: Shapefile, AutoCAD, and a digital elevation model

2. Data cleaning

Make sure your source data is free of problems. This is quite a topic so I won’t go into detail here, but take a look at our multitude of resources, including:

A few useful FME transformers: Intersector, AreaBuilder, Snapper, SliverRemover

3. Data enrichment

Enrichment means adding valuable information to data features. This information can be anything that tells us more about the geometry and attributes — what other features are nearby, how round or oval the feature is, how it’s positioned, etc.

One example is in determining the order that streams flow through a landscape. This can be done with the StreamOrderCalculator transformer, which considers how streams and rivers merge to create a hierarchy and tells you how thick each one should be.

These lines have been thickened based on the calculated stream order.

A few useful FME transformers: anything with ‘Calculator’ in the name (e.g. AreaCalculator), SpatialFilter, NeighborFinder, BoundingBoxReplacer

[ Watch our cartography webinar, where Dmitri and Jen demo all of these steps. ]

4. Generalization

Generalizing a map means deriving simpler polygons and/or a smaller-scale map, so you can visualize an area without including unnecessary details. This process might include simplifying geometries, combining areas, changing geometry types (for example converting polygons to points), modifying geometries to meet requirements, or reducing the number of vertices.

To add a text label to the Philippines, Dmitri created an intermediate polygon with the AreaAmalgamator transformer that merged all of the islands as one.

A few useful FME transformers: Dissolver, Generalizer, AreaAmalgamator, Displacer, CurveFitter

5. Beautification and styling

Here’s where the art comes in. Set the features’ appearances, add labels, and do whatever you need to make it look amazing. For example, use the MapTextLabeller to define rules on how everything should be labelled, or the MapnikRasterizer to convert the map into a nicely styled raster.

Cool styling brought to you by Mapnik.

A few useful FME transformers: anything with ‘Styler’ in the name (e.g. DWGStyler), MapTextLabeller, MapnikRasterizer, RasterHillShader, Labeller

6. Output

The final step is to pick your means of delivery. This might be a CAD or GIS format, PDF file, raster image, a Word or Powerpoint file, or a tile cache on a webpage.

For example, Dmitri made a stylized tiled web map that renders different images as you zoom in.

Tiled web map generated with an FME workspace.

Real Cartographers Using FME

We asked FMEers to send us maps they’ve generated with FME. We got amazing work from Owen Powell, Holly Coxon, Hans van der Maarel, and Joanna Hobbins. Take a look in the below video (extracted from the full webinar).

Tweet this blog

Through this process of integrating, transforming, and styling data, cartographers can more easily achieve the goal of making the world more understandable to everyone. The FME transformers above are just ideas. For more, browse the transformer gallery and cartography tools added to FME Hub by users, and check out our cartography webinar to see demos.

12 Days of FME Hackathon – Results and Recap

 


About FME
|   December 20, 2017   |   By Jen Luther Thomas

Every time I’ve organised a hackathon I’m always so worried that no one will participate. Which is possibly the most irrational fear ever — I’ve never met anyone who didn’t like or want to build cool things with FME.

The same thing happened with the 12 Days of FME Hackathon. Most of the entries didn’t trickle in until the second week, but it was definitely worth the wait (and the number of emails I’d wake up to every day)!

I also think it might be the most viewed question on our Q&A forum, with 2200 views.

I won’t list all of the entries we had — you can check them out for yourself!

Here’s a shout-out to the top 3:

Third Place

Andras Revesz from Barnsley Council submitted the FME Server Log Saver and Timeline Visualiser.

This entry takes the job history of all FME Server jobs and saves them to a database, and uses the Google Charts API to display the workspaces, job number, and when they ran.

Second Place

Dave Eagle from 1Spatial created a FlexNet License Analyser custom transformer, allowing people to keep an eye on how many licenses are checked out and who has them. It’s hard to say if this alone got him to second place — or did the additional photo of a cute puppy help?

This definitely sparked a lot of interest and comments, so hopefully we’ll see something in 2018 uploaded to FME Hub!

First Place

In first place is Sigbjørn Herstad from Norkart who created an un-environmentally friendly process to turn 3D modelled buildings into 2D gingerbread templates. Each part of the 3D building is exploded and put onto one page of a PDF, which you can then print out as a template for a gingerbread house. Perhaps when we move offices in 2018 we could put this to the test!

Note: Paper, bakery and gingerbread ingredients not included.

As the first-place winner, Sigbjørn will receive an FME holiday prize pack. I hope we get some photos of the hat and scarf being worn whilst de-icing a car and holding a lunchbox!

*

Thanks to everyone who participated in the 12 Days of FME Hackathon! Everyone who submitted an entry gets the Holiday Hacker badge and a little boost in reputation (let me know if you’re missing one).

Coders on Couches Drinking Coffee — Augmented Reality

 


About Data
|   December 21, 2017   |   By Stephanie Warner and Tiana Warner

Augmented reality is a powerful new way to understand 3D data. Since FME is so good at transforming data and converting different formats, we wanted to see if we could display 3D objects created by FME in a custom augmented reality app. In this episode of Coders on Couches Drinking Coffee, we chat with Angus, our AR developer, about what he’s been working on.

He’s leveraged ARKit, Apple’s augmented reality framework, which can render 3D objects in OBJ format. When someone wants to use FME to see a 3D object in augmented reality, FME will zip up the OBJ file along with any associated textures, and name the zipfile with the .fmear extension. You can then upload this to the cloud, and use our AR app to see it on an iPhone or iPad.

Coming soon: the plan is to make the app open source on Github, and publish it as a free app on the iOS App Store.

See also:

10 things you (probably) didn’t know the FME Server Experts did in 2017

 


About FME
|   January 4, 2018   |   By Jen Luther Thomas

If I told you I was an ‘FME Server Technology Expert’, would you know what I did every day?

As experts it’s our job to know the product inside and out, so that we can best help our users be successful with FME Server. A lot of this is through helping people on live chat or direct support, but you may have also seen us delivering training, posting on the FME Knowledge Center or presenting webinars.

The FME Server Experts dressed up at Halloween… as FME Server.

2017 was a busy and exciting year, so here are a few other things we did that you may not have known:

1. FME Server Administrator Course

For a long time there’s been the FME Server Authoring course, showing you how to use FME Server. Now, we have the FME Server Administrator course that guides you through installing and configuring FME Server. You can also watch the training by chapter, if you’re interested in a specific topic.

Rylan delivering the FME Server Administrator Course at the FME UC 2017.

2. Active Directory

FME Server Experts worked with customers and developers to improve Active Directory in 2017.1. It’s now configured through the FME Server web interface and allows you to have mixed mode users (Active Directory and FME Server users)!

Active Directory is now configured in the web interface for FME Server 2017.

3. Troubleshooting Guide

In 2017, the FME Server Troubleshooting Guide was redone, making it easier to get troubleshooting tips and find common issues. You can also go to fme.ly/fmeservertroubleshooting for a list of all of the articles available.

4. Installation Recommendations

Ever thought about doing an FME Server installation and wondered which option to choose? We now have an article on the FME Knowledge Center that outlines options and configurations available so you can make an informed decision. You can view that article here.

5. FME Server Tour

One of the first places that someone would look to find out what FME can do and looks like, would be our website.

The FME Desktop Tour had already existed for a while, showing what FME Workbench looked like and how to do the basics, but there wasn’t one for FME Server (until recently).

6. FME Demos

A third of FME Server users don’t know what or where the FME Demos are! Now there’s no excuse: FME Demos previously lived on the FME Server Playground, but now they are on the FME Knowledge Center. Scroll to the bottom of the FME Knowledge Center homepage to find them!

Access FME Demos from the FME Knowledge Center.

7. User Survey

For the last 5 years we’ve put out an FME Server Customer Survey so we can get direct customer feedback. And the end of 2017 we sent out the survey again, this time including questions on FME Desktop, Server and Cloud! As well as wanting to know about the product, we’ve asked users what resources they find most useful when learning and using FME. This means we can focus on creating and improving material that has the most value, to help users get the most out of FME. If you didn’t receive an email asking you to fill out the survey, you can participate:

8. Q&A Forum

Have you asked or answered any questions on the Q&A Forum? The forum is the go-to place for any technical or how-to questions involving FME. Since the summer, FME Experts at Safe have been using the community a lot more, as it’s a great place to share information publicly and answer user questions. Response time averages less than 13 hours for the last six months, and Safers typically respond to over half of the questions posted. There’s also a chance that your question has already been asked and answered!

9. Development Assistance

Experts also get to help out with development assistance. When a new feature is planned for an upcoming release, an expert is assigned to test out the feature and find out what customers really want to do with it. Some of the examples for 2018 are Hadoop HDFS and Version Control.

10. FME Server 2018

So, maybe this one doesn’t really count in a blog post about 2017 but we definitely start planning and preparing for the 2018 release in 2017. There’s lots to do before the release: testing the product, updating training materials and marketing content, preparing FME World Tour presentations. Not to mention supporting our customers through Live Chat, Community Q&A forum and Support Cases. We may be a little quieter at the beginning of the year, but the wait for the FME Server 2018 release will be worth it!

FME World Tour 2018 Registration is Open

 


About FME
|   January 9, 2018   |   By Tiana Warner

Registration is officially open for the FME World Tour 2018! Save your seat to jump on the chance to learn from the experts. Whether you’re new to FME or a life-long user, you can expect a full day of knowledge at this exciting event.

FME World Tour 2018: What’s in it for you?

The Safe Software team embarks on the FME World Tour (FMEWT) every April, visiting 70+ cities around the world in a short timeframe to share all the latest trends and tips for overcoming your top data challenges. Quick facts:

  • Discover what’s new in this FME release and what’s coming next
  • Network with FME users and experts
  • Take home free swag
  • Learn time-saving tips and best practices from Safe’s FME experts, including live demos
  • Hear stories and advice from FMEers during User Presentations
  • Breakfast, lunch, and coffee breaks are included
  • 99.7% of last year’s attendees would recommend the FMEWT to a friend or colleague

Here’s what a typical FME World Tour 2018 day will look like.

Introduction & Getting Started with FME 2018

Kick off the day by meeting the Safe team (four of us usually visit each city) and see what’s new in FME Desktop 2018. We’ll do a live demo of exciting new functionality and give out free swag and resources (like the coveted transformer guides).

Actual photo from what was indisputably the best team on the FME World Tour 2017.

FME Workspace Authoring 2018: Tips and Tricks

This coming release, building FME Workspaces will be a brand new, more efficient experience. Learn how to navigate the new interface and get tips for making your data transformation workflows more powerful than ever.

Redesigned FME Server 2018

FME Server workflows are redesigned for 2018, and this session will focus on how to use the new interface to overcome common data automation challenges.

FME User Presentations: The Data Lifecycle

From planning to prototyping to production, FME is a key part of the data lifecycle in many industries. People from your city who use FME will take the mic to share stories of how they solved their toughest data challenges.

By the way, if you use FME and are attending the FMEWT, we would love to have you present! Your talk can be short and sweet, 10-20 minutes, and you’ll get awesome recognition and swag. Learn more.

FME Around The World

Similar to the above, Safe Software presenters will do a series of lightning talks to showcase how people around the world are using FME to solve their everyday challenges, big and small.

Integrating Data Ecosystems with FME

The afternoon’s technical sessions will hone in on various data ecosystems and how they can work together. You’ll see live demos of how to use FME to integrate and transform various formats. These sessions will include some combination of rasters (satellites, drones, etc), 3D and venue mapping, web data and reporting, and databases. For cities with a dominant industry, we’ll also include a talk focusing on that industry — e.g. Calgary will have a talk focused on using FME for oil and gas, with demos on topics like linear referencing.

Wrap-Up & Prizes

As always, we close with an interactive wrap-up game and a demo to reiterate all you’ve learned today. You’ll get lots of chances to win great prizes!

Data Ecosystems

Ecosystems are the theme for the FMEWT 2018. We looked at all the new formats and functionality going into FME 2018 and noticed a beautiful symmetry in the way data in today’s world is extracted, transformed, and sent to a meaningful output system that enables more data to be gathered and the cycle to continue. It’s like the rainfall cycle, so we built on that visual. We also consider different data systems, like GIS, databases, and business intelligence, to be like ecosystems, rich and diverse and each offering something unique to the world of information exchange.

Something different this year

We’re charging a small fee ($25) for Safe Software-hosted stops, where in past years the FMEWT has been free. This is simply to curb no-shows. A lot of people register for the events each year, and we prepare for that number and bring swag for everyone and book a big enough room and prepare all of the food and… well, some people don’t show up. To avoid wasting resources, to reduce our carbon footprint by minimizing paper and food waste, and to make sure we prepare for the right number of attendees, we’ve added a tiny fee. These fees will go straight back to the community. That is, 100% of the cost will be donated to the local food bank in that city.

*

See you there! Stay tuned for finalized agendas and other updates.

Making Sense of US Census Data With a New FME Transformer

 


About Data
|   January 10, 2018   |   By Nathan Hildebrand

Introducing the USCensusCaller, a new FME Custom Transformer. Here’s how to use it to leverage freely available US Census data, plus how it works, and why this matters.

I have always been interested in demographic and geographic data. I keep a collection of atlases from past eras on my bookshelf just so I can pore over the historical tables, charts, graphs, and maps when I have a spare moment of curiosity. Luckily – moments of nostalgia aside – the technological age has given us Open Data and APIs in place of atlases. A fair trade.

Census data has been on my radar since my undergraduate studies: my urbanism professor once assigned a research project to analyze the demographics of my own neighbourhood in Montreal and I quickly became fascinated with the power of data to inform us about our world.

The Idea: Automatically Grab Census Data via API Calls

Thanks to a recent Safe Software demo development marathon and an idea sparked by fellow Safer Roger Aikema, I had the opportunity to take a look at the US Census data portals and API. The amount of data available is astonishing. As I’m sure many people find, downloading chunks of census data to a local machine is tedious, so with help from colleagues Andrea Eisma and Mark Stoakes, I worked to provide an automated system for grabbing only the desired census data through a series of API calls from within FME.

The Census: Striving for Insights, Not Stockpiles

Data enriching and business intelligence are terms that get thrown around a lot – and for good reason. The amount of data accumulated on a daily basis is becoming incomprehensibly large. Consequently, the focus for decision-makers has to be on harnessing the available data for insights rather than simply stockpiling it. Smart businesses and communities are striving to be data-driven rather than just data rich.

The US Census Bureau has done a lot of stockpiling. And while there are more advanced paid data enriching services out there, the US Census serves their data for free. This sparked the idea for the USCensusCaller.

The database accessed by this transformer selectively compiles variables (somewhat like columns in a table or like attributes of features within FME) from the American Community Survey (ACS) and the US Census. These two datasets are similar, but they differ in their sample size, frequency, and variables. The ACS happens more frequently, but samples less of the population, providing estimates concerning (mostly) economic variables. The US Census happens less frequently but samples more of the population, garnering more precise demographic information. This transformer gives users the choice between the two datasets due in part to differences in variables and in part to some limitations to the API call – more on that below.

The Application: A Hypothetical Business

To get into the mental space of this project, I imagined owning a small chain of coffee shops spread over an urban area, and wanting to know more about my customers. I imagined creating new data in a variety of time-consuming, ineffectual, or costly ways in order to direct a new marketing campaign, for example. Alternatively, I realized I could grab census data for the areas surrounding my stores for free. It seemed like a no-brainer to at least try to tap into the census.

The Technical: Inside the Custom Transformer

Without going too in-depth, I’ll cover the basic processes within the USCensusCaller. At a high level, it is built upon three API calls chained together to output census data for an area based on input points of interest. (A free API key can be provided as a transformer parameter, but is only needed for large or repetitive calls.)

1. Dealing with points

The transformer assumes you have point data in the LL84 (lat/long) coordinate system. Luckily, this is easy to come by in FME as long as you have some kind of location information. A coordinate transformation followed by a coordinate extraction allows the point data to supply the spatial query to TIGERWeb.

2. TIGERWeb query

TIGERWeb is the spatial side of the US Census Bureau. It holds spatial data for all the census geographic areas. Here a FeatureReader grabs the GeoJSON for census blockgroups through the API.

3. Variables

Meanwhile, another API call grabs the complete list of variables for the planning database. These are filtered based on my own generalizations as well as built-in user choice. Unfortunately, the US Census API only supports queries with up to 50 variables, so the hundreds of variables that make up the database schema must be pared down. Finally, a list of variables is built and sent to the next API call.

4. Planning database query

Spatial data from TIGERWeb and the previously chosen variables come together to form the new URL for the data API call.

5. JSON response manipulation

JSON has been the common response throughout the USCensusCaller. Because the response from the US Census API is a JSON array rather than an object, the final extraction is inelegant, but it works! Two JSONExtractors and a JSONFragmenter do the heavy lifting, and after a bit of cleanup, the data is ready for output.

The results are a combination of areas (census blockgroup polygons) and points, which in my example are coffee shops. Each area now has census variables as attributes. These attributes could be further spatially joined to the original stores if desired.

The Future

While taking advantage of open data sources isn’t new or unique to us at Safe Software, I consider this project a small step in a new and interesting direction. With important players like Tableau, Alteryx, Insights, and various data vendors all working in this space, it’s fitting that FME’s versatility would be up to the challenge.

The USCensusCaller is not yet as sophisticated as it could be, but for simple questions like, “Which of my potential sites is the best place to put a new trendy coffee shop based on local income, real estate cost, resident age, and education?” or even “What is the demographic makeup of my neighbourhood?” it is an excellent starting point.

P.S. A similar solution is currently unavailable for our Canadian friends. But perhaps not forever as changes are made to the StatCan API.

FME 2018.0 Release: Top 23 New Features

 


About FME
|   January 16, 2018   |   By Tiana Warner

FME 2018.0 Betas are available, with the official release coming in early March.

There’s a lot to be excited about this release! Our developers have worked hard to make workspace authoring more efficient, to add new formats and transformers, and to completely refresh how you automate your workflows in FME Server. Here are the top 23 new features. For a detailed list of what’s new, check the What’s New in FME Desktop 2018.0 and What’s New in FME Server 2018.0 documentation.

Connect & Transform – More format support and transformation power

1. New Format Reader/Writers

  • Apple Venue Format Reader & Writer
  • Bentley i-Model Reader (V1)
  • CALS (Continuous Acquisition and Life-Cycle Support) Raster Reader & Writer
  • Esri Geodatabase Mosaic Dataset Writer (ArcSDE & File Geodatabase)
  • Esri Indexed 3D Scene Layer Writer (i3S)
  • GDAL Generic Raster Reader (read anything GDAL supports)
  • 54 new GDAL-based Raster Formats
  • GDAL VRT Raster Reader
  • Google Cloud Spanner Reader & Writer
  • HYPACK Reader & Writer
  • IDRISI Raster Reader & Writer
  • LDAP (Active Directory) Reader
  • Microsoft PowerPoint Writer
  • Microsoft Word Writer
  • PCD (Point Cloud Data) Reader & Writer
  • PDF Reader (2D/Raster/Text)
  • SOSI GML Reader & Writer
  • Well Log ASCII Standard (WLAS) Reader

Fun fact: this release marks a milestone — FME now supports over 400 formats!

2. New Transformers

  • AreaGapAndOverlapCleaner
  • HDFSConnector (for moving files into and out of Hadoop)
  • FeatureJoiner
  • MeshSimplifier
  • MSWordStyler
  • PowerPointStyler
  • ProjectWiseConnector
  • S3Connector
  • SalesforceConnector
  • SurfaceDissolver

Plus, browse FME Hub for hundreds of transformers added by FME users and the experts at Safe Software.

3. Indoor Mapping Formats

Connect to indoor mapping applications with added support for Apple Venue Format (reader & writer) and IBM TRIRIGA (transformers on FME Hub).

Indoor map of YVR airport, viewed in the FME Data Inspector.

4. PDF Reading

PDF is no longer where data goes to die. FME’s new PDF Reader means you can extract rasters, vector data, and text from PDF documents.

Left: PDF data viewed in the FME Data Inspector. Right: PDF data viewed in Adobe Reader. Not too shabby!

5. Create 3D Web Visualizations

Create 3D visualizations that anyone can view in a web browser, with new support for Esri I3S and glTF for Cesium (FME 2017).

Left: every building in NYC loaded as Cesium 3D Tiles. Right: with textures added, written to I3S and loaded in ArcGIS Earth.

6. Generate Microsoft Excel, Word, and PowerPoint Files

New transformers let you style and output your data for Excel spreadsheets, Word pages, and PowerPoint slides.

A map of Canada written to PowerPoint with FME. Each province is its own polygon with a label.

7. Coordinate System Updates

Including:

  • Australia’s GDA 2020 & German Niedersachsen datum shift grids available
  • RSO System Updates (Malaysia)

8. More Format and Transformer Improvements

We’ve improved a lot of existing formats and transformers. For example:

  • S3 transformers have been merged into a single S3Connector, and files can be accessed directly like a filesystem.
  • Creator now supports creating clothoids and 3D arcs.
  • FeatureWriter supports output ports.
  • Bufferer can now buffer 3D objects.
  • Several transformers have an added Tolerance parameter.

FeatureWriter with multiple input and output ports.

For specifics, see the What’s New doc.

Work Play More Efficiently – FME Workbench enhancements

9. Collapsible Bookmarks

More easily manage large workspaces by collapsing sections under colourful bookmarks that you can expand when necessary.

10. Partial Workspace Runs (Tech Preview)

Get immediate feedback when authoring FME Workspaces by running only the portion of the workspace you’re focused on.

“Run from This”, “Run just This”, and “Run to This” buttons.

11. Embedded FME Data Inspector (Tech Preview)

Instantly view transformation results right inside FME Workbench.

12. Copy & Paste Transformer Test Conditions

Copy-paste conditions and conditional values between transformer dialogs.

13. Customized Translation Log

Filter the translation log to display only error or warning messages.

14. Better Performance

Significant improvements in engine performance with many scenarios clocking in 30% faster and using 45% less memory.

15. FME Python API

A documentation overhaul makes the FME Python API much easier to navigate and learn from.

FME Server Updates

FME Server 2018.0 was built from the ground up and has a lot of new features to get excited about.

16. Job Filtering

In the Job interface, flip a switch to filter by jobs that have succeeded or failed.

17. Password Recovery

Administrators can set up a system email account so users can recover their passwords.

18. Dark Mode

Customize your FME Server interface.

19. Workspace Viewing (Tech Preview)

View your FME Workspace from directly within the FME Server interface.

20. Deployment Visualization

See the hosts comprising a deployment and the server components on each host.

21. Project Sharing

Share FME Server Projects without sharing security credentials. Coming soon to FME Hub.

22. FME Server on Docker

This is out of tech preview. Quickly deploy FME Server on any platform that supports Docker.

23. Version Control

Also out of tech preview. See previous versions in the FME Server interface, and create a version history when publishing a workspace from FME Workbench.

Bonus: FME Augmented Reality

Soon you’ll be able to convert 3D objects to .fmear and view them in augmented reality devices. Learn more.

*

Stay tuned for the official release in early March!

  • Check our webinars page for a recording of today’s Unveiling webinar with Don and Dale, and to sign up for the upcoming Deep Dive into FME Desktop and FME Server webinars.
  • Sign up for the FME World Tour to see live demos and learn tips and tricks from the experts.

Elastic Processing in the Cloud with Docker and FME Server

 


About FME
|   January 18, 2018   |   By Don Murray and Grant Arnold

Cloud and container technologies are key to efficiently completing compute-intensive tasks. We have been experimenting with “Elastic Processing” for large tasks that can be broken into many smaller ones.

Here’s how we used FME Server to exploit the elasticity of the cloud. Plus, see how we put it to the test and cut an 8-hour tile cache workflow down to 20 minutes.

Elastic Processing Pattern

The pattern for this workspace is very simple, with only step 3 varying with the Task being performed.

Elastic Pattern FME Workspace

Download sample workspace

This Elastic Processing Pattern is great for tasks that can be broken down into a large number of small jobs. The pattern is as follows:

  1. Define the job queue. All jobs for the task will be submitted to this FME Server queue. The job queue name must be unique for the task. This is done via the FMEServerJobQueueCreator transformer on FME Hub.
  2. Launch Compute Resources.  The number of hosts and engines to launch are specified.  The job queue created in step 1 is also specified. As of FME 2018, engines can be launched with one or more job queue names specified. The engines will then only process jobs from those queues. In this workflow, we specify 1 queue. The FMEServerScalerRancher transformer launches the specified machines and engines, and returns immediately. It doesn’t have to wait for them to be launched but rather returns as soon as it can.
  3. Submit and Process Jobs. The jobs that make up the task are all submitted to the FME Server. Again the job queue is specified so every job goes to the job queue and is processed by one of the engines assigned to the job queue and launched in Step 2. The FMEServerJobSubmitter is set to Submit Jobs -> In Parallel and Wait for jobs to Complete-> Yes.  This ensures the job waits until all jobs are complete. Note: We are not totally happy with this as we are using an engine that is waiting for the work to finish.  We will address this in a future version of FME Server. 😉
  4. Release the Compute Resources. After the jobs are completed we immediately scale the engines again using the FMEServerScalerRancher transformer.  This time we specify 0 engines and 0 machines.
  5. Delete the job queue.  This step is not necessary but we are being nice to our server so we don’t leave unused job queues laying around. This is done via the FMEServerJobQueueDeleter transformer on FME Hub.

Supporting Technology

To make this happen, we relied on leading-edge technology upon which we are building future FME Server products.

  • Docker – FME Server is now deployable as a collection of containers. These containers give us the power to easily deploy FME Server in new and powerful ways, such as this pattern. Suffice it to say that containers are going to play a bigger part in FME Server’s future. In this example, all FME Server including the FME Engine components are running in Docker.
  • AWS – Amazon Web Services compute instances. Here we used Amazon EC2 instances. This pattern, however, could easily be deployed on any cloud stack such as Azure or Google. Indeed we will see more cloud offerings and better container support from cloud providers.
  • Rancher – Host management layer that enables us to launch new compute nodes (EC2 in this case) and also the FME Engine containers that will do the work.
  • FME Server 2018 – New capabilities of FME Server 2018 help make this possible. The new job queue capability is one good example.
  • FME Hub – FMEServerJobQueueCreator and FMEServerJobQueueDeleter. These two transformers make it easy to create and delete FMEServer job queues. FME Hub is in its infancy. Look for this to grow by leaps and bounds over the next year. I check this weekly and recommend you do too. Amazing how FME is being expanded by the FME community.

Architectural view of supporting technology.

Watch: Elastic Processing Pattern in operation

Grant and I made a video showing this pattern in operation.  You will see the Rancher interface, scaling, and the whole workflow.

Putting the Pattern to Work

Having built the technology, we didn’t have to look far for a sample Task on which to give it a try. We simply visited our local FME Scenario Guru, Dmitri. To say he was excited is an understatement. We put this to the test on generating a raster tile cache for a large area.

Large Area Raster Tile Cache Generation

Tile caches of large areas can have hundreds of thousands or even millions of tiles. Generating them is a compute-intensive task that can take days on a single machine. Running this on a desktop resulted in the machine being so slow that Dmitri couldn’t even play solitaire!

Modern imagery providers such as Planet now deliver daily updates for the entire planet. The only way to generate Tile caches in a timely way for large areas is to leverage the power of many machines.

With the technique described here, we can now use as many machines and engines as we want to process the data in a timely fashion.

In our tests, we used an RDS-based PostGIS database preloaded with the OSM datasets covering a large metropolitan area.

The tile cache for London for up to zoom level 17 consists of ~800,000 tiles, and takes well over 8 hours on a single machine. Launching 24 machines using this Elastic Pattern, we get the same result in about 20 minutes.

Large tile cache for London. Be sure to zoom in to get an idea of the extent. Thanks, Dmitri!

Cloud and container technologies are separately very exciting, as they promise to change the way resources are accessed and applications are deployed. Together, the possibilities are even more exciting. At Safe, we are incorporating and leveraging both of these technologies to enable FME to be deployed in ways that before we could only dream of.

We would very much like to hear your thoughts on this technology and tasks, and where you see it being valuable. We are also always happy to chat if, like us, you are excited by the possibilities shared here.

Syncing SaaS with FME

 

Small businesses have gotten SaaS-y! And with good reason. These cloud-hosted solutions execute vital functions at a fraction of the cost of on-site applications. With each having an area of specialization, several are often in use in one organization. Overlap of business processes and information between them necessitates data syncing. Without an integration solution, this would have to be done manually — a huge expense of time, and therefore money.

Challenge: Automated Integration of Freshbooks and Quickbooks

Kevin Weller of Modern geoSystems wanted to eliminate duplicate data entry tasks across SaaS with an automated integration process. Specifically, he wanted to integrate Freshbooks — used for billing, and Quickbooks — used for general accounting. He also had plans to incorporate spatial analysis into the process.

He evaluated several ETL platforms, but kept running into an issue with spatial integration. To accomplish the complex spatial tasks he desired, these tools depend on a combination of external GIS tools, services and spatial databases. Web-based API integration platforms were too glitchy, and he couldn’t control the required processes in fine enough detail.

Achieving Fine-Tuned Integration with FME’s Flexibility

A certified FME professional with years of experience using the platform to solve GIS-related integration challenges, Kevin was confident he already had the solution to Modern geoSystems’ problem in his hands. FME’s flexible tools like the HTTPCaller — which connects to any web service with an API— enabled the fine-tuned integration he desired in addition to having the most advanced spatial data manipulation tools available.

The first iteration of this integration was brilliantly simple. Using FME Desktop to build workflows and FME Cloud to automate, Kevin configured a process that connects to Freshbooks via its API and looks for non-draft invoices from a specified date forward. It then copies the invoices to Quickbooks Online using its API. The process also updates clients and terms lists from QuickBooks and copies the invoices into the company’s Operational Data Store that houses the official standard of all company data. He also uses the Notification Service on FME Cloud to keep the Operational Data Store up-to-date as data changes in the source systems.

Kevin has successfully established FME as a central integration hub for Freshbooks, Quickbooks and other enterprise systems. He is confident that it can handle any new processes or applications that come along and is currently adding integrations with other SaaS like Twitter, LinkedIn, Google Analytics, Qlik, Slack and AWS.

Another flexibility feature of FME that helps users fine tune their integrations is its ability to integrate with programming languages when necessary. Kevin was able to build a custom transformer – using Python and SQLite for its internal logic – that sorts some datasets faster than FME’s own built-in Sorter. The SQLiteSorter is available for download on FME Hub.

So there you have it. Another case of FME’s flexibility being implemented to sync data and business processes across SaaS and enterprise systems. It has saved Modern geoSystems time with its automation and money that might have been spent on third-party ETL software.

Let us know how you’re using FME to connect to and integrate SaaS at your organization!

How to Join Data with FME and (Maybe) Land Your Dream Job!

 


About FME
|   January 30, 2018   |   By Mark Ireland

Inner Join? Is that the opposite of an Outer Join?

When I was interviewed for a job with the team at Safe Software, one question asked was “Do you know the difference between a left join and an inner join?”

Although I’d used SQL and joins quite a lot before, I couldn’t remember the exact definition and mumbled an answer. I was hired, so the response must have been more convincing than I thought. Either that or they really liked my joke about the cinnamon buns.

The thing is, because I’ve had access to FME, I’ve rarely since needed to do SQL joins; so my knowledge of left and inner joins has remained a bit sketchy… until today!

Because today is when I started to try out the latest transformer to hit the Filters and Joins category: The FeatureJoiner.

What is the FeatureJoiner?

So the FeatureJoiner is a transformer that joins data together. It most closely resembles the FeatureMerger (which it may eventually replace). Here it is on the canvas:

FeatureJoiner on the FME Workbench canvas

This workspace joins Facility features with PostalAddress records. You can see that instead of Requestor/Supplier the transformer has Left/Right ports, and one fewer output port; but otherwise it really does look like the FeatureMerger.

However, a look at the parameters dialog shows a lot more differences:

FeatureJoiner parameter dialog

The Join On, Conflict Resolution, and Geometry Handling parameters are fairly obvious, and are similar to those that already exist in the FeatureMerger. But the key parameter is Join Mode…

Join Modes

The Join Mode is where it becomes necessary to know about Left Joins and Inner Joins. That’s because the three modes are Left Join, Inner Join, and Full Join.

For our training materials I made up a table that shows what these modes do:

Mode Description Depiction Joined Output Unjoined Left Unjoined Right
Left Left features look for a match and are output whether they find a match or not All matches plus unmatched Left features None Unused Right features
Inner Left features look for a match and are output if they find one All matches only Unmatched Left features Unused Right features
Full Both Left and Right features output through the Joined output port, whether they find a join or not All matches plus unmatched Left and Right features None None

The simplest way to understand the operation is this: the overlapping part in each diagram is always output. It’s a join so the features are output through the Joined port.

The Left/Inner/Full parameter controls not the joined features, but the features that aren’t joined.

  • In Inner mode features without a join exit through either the Unjoined Left or Unjoined Right port.
  • In Left mode, Left features without a match still exit through the Joined port. Right features without a match exit the Unjoined Right port.
  • In Full mode, both Left and Right features without a match still exit through the Joined port.

The Facility Example

So, remember our Facility/Address example, which matches Facility features to Address records:

FeatureJoiner example join

In Inner mode, only facilities with a matching address exit through the Joined port. It’s a good way to find non-matching features, which exit through the Unjoined Left port.

But in Left mode, all facilities exit the Joined port, whether they have a match or not. It’s not so good for data QA, but it’s fine if we know not every facility has a matching address.

In Full mode, the Joined output would include ALL addresses, used or not. It’s probably not needed in this case, but it is useful in other scenarios.

Now, you may have noticed that the FeatureJoiner has no equivalent to the FeatureMerger “handle duplicate suppliers” parameter. To understand that you need to understand what we mean by a “match”…

Multiple Join Matches

To understand this, let’s take a look at this with some feature count numbers included:

FeatureJoiner - example join with feature counts

How many features will exit the Joined port? Well, it depends.

If we assume a strict 1:1 match between each Facility and an Address, then 8 features will exit as Joined. Why? Because we have 8 matches. The count will be thus:

Mode Joined Unjoined Left Unjoined Right
Left 8 0 1,463,434
Inner 8 0 1,463,434
Full 1,463,442 0 0

The Full Joined number of 1,463,442 consists of 8 matches and 1,463,434 unmatched right features.

So that’s good. However, we might not have such a clean 1:1 match between Facility and Address table. We could have 1:M, M:1, or even M:M.

Let’s say each Facility has two matches in the Address table. Then we get this:

Mode Joined Unjoined Left Unjoined Right
Left 16 0 1,463,426
Inner 16 0 1,463,426
Full 1,463,442 0 0

Why are we getting more features out than we put into the Left port? Because we are getting one feature per match, and there are 16 matches!

This is not something the FeatureMerger would do. It would either ignore the second match or create a list. But this is designed to be a SQL equivalent, and so it works differently. In fact – in an extreme case – if we assumed that every Facility has AddressID=1, and every Address has AddressID=1, then every facility would match to every address!

That would give us 8 x 1,463,442 = 11,707,536 matches!

To put it another way, in Left or Inner mode we could put in 8 “requestor” features and get out 11,707,536. It’s not wrong, just perhaps different than what you’re used to.

Output Order

As a bit of an advanced topic, matches exit the the transformer in the same order as the features enter the Left port.

So if the Left features are sorted in a particular order when they enter the transformer, that order is unchanged in the output, for either a Left or Inner join.

If you want to retain the input order for a Full join, then both the Left and Right inputs must be already sorted in order.

Which to Use?

So which transformer should you be using? Well there are two aspects to consider: functionality and performance.

For functionality the FeatureJoiner is certainly nicer for users used to database terminology and functionality. It’s designed to mimic SQL so the results should be the same as if you’d entered the same SQL join commands.

The FeatureJoiner is fast like a sports car!The key output difference is the one-feature-per-match concept of the FeatureJoiner. The FeatureMerger would let you create a list instead. But I’d say that a lot of the time you immediately use a ListExploder to break the list up into features anyway, so I’m not sure how much disadvantage it is to not have a List parameter in the FeatureJoiner.

Plus, if you really need a list, follow up with a ListBuilder grouping by your join keys.  This is where output order is important, as you might be able to set Input Ordered by Group in the ListBuilder, if the FeatureJoiner input was suitably sorted.

As for performance, the FeatureJoiner boasts an improved design, and gets an added boost by using the “Feature Table” technology that speeds up so many other readers, writers, and transformers. So if performance is important then you should definitely try out the FeatureJoiner.

Summary

You may be more comfortable sticking with the familiar FeatureMerger at first, but it’s worth going out of your comfort zone to at least try the FeatureJoiner.

You might even learn some terminology to help land your dream job!

As an example of the improved performance, check out the video of the FME 2018 unveiling:

There (jump to 58 minutes if it doesn’t automatically) Don and Dale carry out a left join on a CSV file of 320,000 records. The workspace takes 53 seconds to complete with the Joiner and 5 minutes, 33 seconds to complete with the FeatureMerger. So performance improves greatly in that example.

Incidentally, notice in the video that their transformer’s name is the Joiner. We’ve since changed it to FeatureJoiner, to be less confusing (given that there used to be a Joiner, which is now the DatabaseJoiner). I believe Joiner is still an alias for the DatabaseJoiner, but “DJ” or “FJ” in Quick Add bring up these transformers quicker anyway!

Additionally, if you want to carry out joins, don’t forget the InlineQuerier transformer. It constructs an ad hoc database and allows you to carry out SQL commands on it – so if your join is more complex than the FeatureJoiner/Merger allow, try the InlineQuerier instead.

NB: If you are looking for a job(!) follow FMEJobs on Twitter, or check out the Safe Software careers page – and don’t forget to brush up on joins before your interview!