Specifically...

How do we work with the
NASA NEX Down-sampled Climate Projections (NEX-DCP30)
open data set?

Global Circulation Models

Models for predicting world temperature and precipitation.

IPCC Assessment Report

• IPCC = Intergovernmental Panel on Climate Change
• Assessment Report 5 (AR5) published in 2014.
• More than 800 authors

3 Key Categories:

• Model

  • 33 different models
  • Model Ensembling

• Dataset

  • Temperature MAX
  • Temperature MIN
  • Precipitation

• Scenario

  • Historical
  • Future RCPs

Representative Concentration Pathways

NEX Down-sampled Data

• Monthly data over conterminous US
  • Historical from 1950 - 2006
  • 4 RCP scenarios from 2006 - 2099
• 8190 netCDF files on S3 - s3://nasanex/NEX-DCP30
• 15.3 TB in compressed GeoTiff tiles.
• RCP 8.5, max for datatype/model combo: 90.92 GB

• Scala library for doing all things geospatial.
• framework for doing distributed raster processing on Akka and Spark.
• Includes local, zonal, focal, and global operations on rasters.
• Currently in incubation at

• Fast and general engine for large-scale data processing
• Does things Hadoop doesn't, like cache intermediate results in memory.
• Written in Scala!
• Also has bindings for Python and Java

Accumulo

• Big table implementation
• Has sorted indexing
• Columnar database
• Also used by GeoMesa, another Scala project at LocationTech

Tiles

Tiles

Indexing tiles

RasterRDD[K]

• SpatialKey
• TemporalKey
• SpaceTimeKey

Step 1:

Export the netCDF data into 512x512 GeoTiff tiles.

• Python code using GDAL and rasterio.
• AWS Auto scaling groups and SQS.
• Code: https://github.com/lossyrob/nex-chunker-worker

Step 2:

Ingest the data into Accumulo using GeoTrellis-Spark.

• Ingest the GeoTiffs to Accumulo in parallel across a cluster.
• Ingest consists of
  • reprojection
  • mosaicing to tile scheme (TMS)
  • pyramiding up zoom levels
  • Calculate index splits.