less than 1 minute read

earthmover

earthmover is a command-line data transformation tool I’ve built as part of my work at Education Analytics. It

  • loads data from a variety of sources, including CSV and other types of files, relational database connections, and more
  • transforms the data according to instructions in a YAML file
  • renders a Jinja template (which can be any text-based data format, including JSON, XML, HTML, YAML, and more) for each row of transformed data, and saves the output to a file

earthmover is similar in some ways to dbt, but it is the transformation execution engine (rather than issuing SQL commands to a database backend which is the execution engine). earthmover is built using a number of Python libraries including Dask and NetworkX.

At EA, we use earthmover to transform various types of flat files into JSON according to the Ed-Fi data standard, which we then send to Ed-Fi APIs using lightbeam. You can learn more about both tools in this presentation.

You May Also Enjoy

Resistance and momentum

4 minute read

I’ve been thinking recently about workplace dynamics in terms of concepts from classical mechanics such as resistance (and related terms like friction or dra...

Working from home

1 minute read

With the spread of COVID-19 driving many more people to work from home, I thought I’d share some of my tips for working from home (based on my 10+ years of e...

Back from Europe

less than 1 minute read

I haven’t posted since April, which is because my wife and I had the opportunity to travel through Europe for 10 weeks! If you’d like, you can read more abou...