landsat-util

https://travis-ci.org/developmentseed/landsat-util.svg?branch=v0.5 Downloads License

Landsat-util is a command line utility that makes it easy to search, download, and process Landsat imagery.

This tool uses Development Seed’s API for Landsat Metadata.

This API is accessible here: https://api.developmentseed.org/landsat

You can also run your own API and connect it to this tool.

Table of Contents:

Installation

Mac OSX

$: pip install landsat-util

Ubuntu 14.10

Use pip to install landsat-util. If you are not using virtualenv, you might have to run pip as sudo:

$: sudo apt-get update
$: sudo apt-get install python-pip python-numpy python-scipy libgdal-dev libatlas-base-dev gfortran libfreetype6-dev
$: pip install landsat-util

Other systems

Make sure Python setuptools is installed:

$: python setup.py numpy six
$: python setup.py install

Docker

If you have docker installed, you can use landsat-util image on docker:

$: docker pull developmentseed/landsat-util
$: docker run -it developmentseed/landsat-util:latest /bin/sh -c "landsat -h"

To use docker version run:

$: docker run -it -v ~/landsat:/root/landsat developmentseed/landsat-util:latest landsat -h

Example commands:

$: docker run -it -v ~/landsat:/root/landsat developmentseed/landsat-util:latest landsat search --cloud 4 --start "january 1 2014" --end "january 10 2014" -p 009,045
$: docker run -it -v ~/landsat:/root/landsat developmentseed/landsat-util:latest landsat download LC80090452014008LGN00 --bands 432

This commands mounts landsat folder in your home directory to /root/landsat in docker. All downloaded and processed images are stored in ~/landsat folder of your computer.

If you are using Windows replace ~/landsat with /c/Users/<path>.

Upgrade

$: pip install -U landsat-util

If you have installed previous version of landsat using brew, first run:

$: brew uninstall landsat-util

Running Tests

$: pip install -U requirements/dev.txt
$: nosetests

Or:

$: python setup.py test

Overview: What can landsat-util do?

Landsat-util has three main functions:

  • Search for landsat tiles based on several search parameters.
  • Download landsat images.
  • Image processing and pan sharpening on landsat images.

These three functions have to be performed separately.

Help: Type landsat -h for detailed usage parameters.

Download

You can download tiles using their unique sceneID, which you get from landsat search.

Landsat-util will download a zip file that includes all the bands. You have the option of specifying the bands you want to down. In this case, landsat-util only downloads those bands if they are available online.

Examples of download:

Download images by their custom sceneID, which you get from landsat search:

$: landsat download LC80090452014008LGN00

Download only band 4, 3 and 2 for a particular sceneID:

$: landsat download LC80090452014008LGN00 --bands 432

Download multiple sceneIDs:

$: landsat download LC80090452014008LGN00 LC80090452015008LGN00 LC80090452013008LGN00

Image processing

You can process your downloaded tiles with our custom image processing algorithms. In addition, you can choose to pansharpen your images and specify which bands to process.

Examples of image processing:

Process images that are already downloaded. Remember, the program accepts both zip files and unzipped folders:

$: landsat process path/to/LC80090452014008LGN00.tar.bz

If unzipped:

$: landsat process path/to/LC80090452014008LGN00

Specify bands 3, 5 and 1:

$: landsat process path/to/LC80090452014008LGN00  --bands 351

Process and pansharpen a downloaded image:

$: landsat process path/to/LC80090452014008LGN00.tar.bz --pansharpen

A note on returned products

Scenes acquired after 2015 will be downloaded from AWS Public Data Sets while scenes acquired before 2015 will be downloaded from Google Earth Engine. AWS provides the bands separately and so landsat-util will also pass along the bands individually if requested. In the case of Google Earth Engine, only the full, compressed image bundle is available (including all bands and metadata) and will be downloaded no matter what bands are requested.

To Do List

  • Add Sphinx Documentation
  • Add capacity for NDVI output
  • Add alternative projections (currently only option is default web-mercator; EPSG: 3857)
  • Connect search to Google Address API
  • Include 16-bit image variant in output
  • Add support for color correct looping over multiple compressed inputs (currently just 1)

Module Index

downloader.py

class landsat.downloader.Downloader(verbose=False, download_dir=None)[source]

Bases: landsat.mixins.VerbosityMixin

The downloader class

amazon_s3(scene, band, path)[source]

Amazon S3 downloader

Parameters:
  • scene (String) – The scene ID.
  • band (String, Integer) – The band number.
  • path (String) – The directory path to where the image should be stored
Returns:

Boolean

amazon_s3_url(sat, filename)[source]

Return an amazon s3 url the contains the scene and band provided.

Parameters:
  • sat (dict) – Expects an object created by scene_interpreter method
  • filename (String) – The filename that has to be downloaded from Amazon
Returns:

(String) The URL to a S3 file

download(scenes, bands=None)[source]

Download scenese from Google Storage or Amazon S3 if bands are provided

Parameters:
  • scenes (List) – A list of scene IDs
  • bands – A list of bands. Default value is None.
Returns:

(List) includes downloaded scenes as key and source as value (aws or google)

fetch(url, path, filename)[source]

Downloads the given url.

Parameters:
  • url (String) – The url to be downloaded.
  • path (String) – The directory path to where the image should be stored
  • filename (String) – The filename that has to be downloaded
Returns:

Boolean

get_remote_file_size(url)[source]

Gets the filesize of a remote file.

Parameters:url (String) – The url that has to be checked.
Returns:int
google_storage(scene, path)[source]

Google Storage Downloader.

Parameters:
  • scene (String) – The scene id
  • path (String) – The directory path to where the image should be stored
Returns:

Boolean

google_storage_url(sat)[source]

Returns a google storage url the contains the scene provided.

Parameters:sat (dict) – Expects an object created by scene_interpreter method
Returns:(String) The URL to a google storage file
remote_file_exists(url)[source]

Checks whether the remote file exists.

Parameters:url (String) – The url that has to be checked.
Returns:True if remote file exists and False if it doesn’t exist.
scene_interpreter(scene)[source]

Conver sceneID to rows, paths and dates.

Parameters:scene (String) – The scene ID.
Returns:dict
Example output:
>>> anatomy = {
        'path': None,
        'row': None,
        'sat': None,
        'scene': scene
    }
exception landsat.downloader.IncorrectSceneId[source]

Bases: exceptions.Exception

Exception to be used when scene id is incorrect

exception landsat.downloader.RemoteFileDoesntExist[source]

Bases: exceptions.Exception

Exception to be used when the remote file does not exist

uploader.py

image.py

landsat.py

mixins.py

class landsat.mixins.VerbosityMixin[source]

Bases: object

Verbosity Mixin that generates beautiful stdout outputs.

exit(message)[source]

outputs an exit message and exits

Parameters:message (String) – The message to be outputed
Returns:void
output(value, normal=False, color=None, error=False, arrow=False, indent=None)[source]

Handles verbosity of this calls. if priority is set to 1, the value is printed

if class instance verbose is True, the value is printed

Parameters:
  • value (String) – a string representing the message to be printed
  • normal (String) – if set to true the message is always printed, otherwise it is only shown if verbosity is set
  • color – The color of the message, choices: ‘red’, ‘green’, ‘blue’
  • error (Boolean) – if set to true the message appears in red
  • arrow (Boolean) – if set to true an arrow appears before the message
  • indent (Boolean) – indents the message based on the number provided
Returns:

void

subprocess(argv)[source]

Execute subprocess commands with proper ouput. This is no longer used in landsat-util

Parameters:argv (List) – A list of subprocess arguments
Returns:void
verbose = False

search.py

class landsat.search.Search[source]

Bases: object

The search class

cloud_cover_prct_range_builder(min=0, max=100)[source]

Builds cloud cover percentage range query.

Parameters:
  • min (float) – float specifying the minimum percentage. Default is 0
  • max (float) – float specifying the maximum percentage. Default is 100
Returns:

String

date_range_builder(start='2013-02-11', end=None)[source]

Builds date range query.

Parameters:
  • start (String) – Date string. format: YYYY-MM-DD
  • end (String) – date string. format: YYYY-MM-DD
Returns:

String

lat_lon_builder(lat=0, lon=0)[source]

Builds lat and lon query.

Parameters:
  • lat (float) – The latitude. Default is 0
  • lon (float) – The The longitude. Default is 0
Returns:

String

query_builder(paths_rows=None, lat=None, lon=None, start_date=None, end_date=None, cloud_min=None, cloud_max=None)[source]

Builds the proper search syntax (query) for Landsat API.

Parameters:
  • paths_rows (String) – A string in this format: “003,003,004,004”. Must be in pairs and separated by comma.
  • lat (String, float, integer) – The latitude
  • lon (String, float, integer) – The The longitude
  • start_date (String) – Date string. format: YYYY-MM-DD
  • end_date (String) – date string. format: YYYY-MM-DD
  • cloud_min (float) – float specifying the minimum percentage. e.g. 4.3
  • cloud_max (float) – float specifying the maximum percentage. e.g. 78.9
Returns:

String

row_path_builder(path='', row='')[source]

Builds row and path query.

Parameters:
  • path (String) – Landsat path. Must be three digits
  • row (String) – Landsat row. Must be three digits
Returns:

String

search(paths_rows=None, lat=None, lon=None, start_date=None, end_date=None, cloud_min=None, cloud_max=None, limit=1)[source]

The main method of Search class. It searches Development Seed’s Landsat API.

Parameters:
  • paths_rows (String) – A string in this format: “003,003,004,004”. Must be in pairs and separated by comma.
  • lat (String, float, integer) – The latitude
  • lon (String, float, integer) – The The longitude
  • start_date (String) – Date string. format: YYYY-MM-DD
  • end_date (String) – date string. format: YYYY-MM-DD
  • cloud_min (float) – float specifying the minimum percentage. e.g. 4.3
  • cloud_max (float) – float specifying the maximum percentage. e.g. 78.9
  • limit (integer) – integer specigying the maximum results return.
Returns:

dict

Example:
>>> search = Search()
>>> search('003,003', '2014-01-01', '2014-06-01')
>>> {
        'status': u'SUCCESS',
        'total_returned': 1,
        'total': 1,
        'limit': 1
        'results': [
            {
                'sat_type': u'L8',
                'sceneID': u'LC80030032014142LGN00',
                'date': u'2014-05-22',
                'path': u'003',
                'thumbnail': u'http://....../landsat_8/2014/003/003/LC80030032014142LGN00.jpg',
                'cloud': 33.36,
                'row': u'003
            }
        ]
    }

utils.py

class landsat.utils.Capturing[source]

Bases: list

Captures a subprocess stdout.

Usage:
>>> with Capturing():
...     subprocess(args)
landsat.utils.check_create_folder(folder_path)[source]

Check whether a folder exists, if not the folder is created.

Parameters:folder_path (String) – Path to the folder
Returns:(String) the path to the folder
landsat.utils.convert_to_integer_list(value)[source]

Converts a comma separate string to a list

Parameters:

value (String) – the format must be 003,003,004,004 (commas with no space)

Returns:

List

Example:
>>> convert_to_integer_list('003,003,004,004')
['003', '003', '004', '004']
landsat.utils.create_paired_list(value)[source]

Create a list of paired items from a string.

Parameters:

value (String) – the format must be 003,003,004,004 (commas with no space)

Returns:

List

Example:
>>> create_paired_list('003,003,004,004')
[['003','003'], ['004', '004']]
landsat.utils.exit(message, code=0)[source]

output a message to stdout and terminates the process.

Parameters:
  • message (String) – Message to be outputed.
  • code (int) – The termination code. Default is 0
Returns:

void

landsat.utils.georgian_day(date)[source]

Returns the number of days passed since the start of the year.

Parameters:

date (String) – The string date with this format %m/%d/%Y

Returns:

int

Example:
>>> georgian_day('05/1/2015')
121
landsat.utils.get_file(path)[source]

Separate the name of the file or folder from the path and return it.

Parameters:

path (String) – Path to the folder

Returns:

(String) the filename

Example:
>>> get_file('/path/to/file.jpg')
'file.jpg'
landsat.utils.get_filename(path)[source]

Return the filename without extension.

Parameters:

path (String) – Path to the folder

Returns:

(String) the filename without extension

Example:
>>> get_filename('/path/to/file.jpg')
'file'
landsat.utils.reformat_date(date, new_fmt='%Y-%m-%d')[source]

Returns reformated date.

Parameters:
  • date (String) – The string date with this format %m/%d/%Y
  • new_fmt – date format string. Default is ‘%Y-%m-%d’
Returns:

int

Example:
>>> reformat_date('05/1/2015', '%d/%m/%Y')
'1/05/2015'
landsat.utils.three_digit(number)[source]

Add 0s to inputs that their length is less than 3.

Parameters:

number (int) – The number to convert

Returns:

String

Example:
>>> three_digit(1)
'001'
class landsat.utils.timer[source]

Bases: object

A timer class.

Usage:
>>> with timer():
...     your code
landsat.utils.year(date)[source]

Returns the year.

Parameters:

date (String) – The string date with this format %m/%d/%Y

Returns:

int

Example:
>>> year('05/1/2015')
2015

Important Notes

  • All downloaded and processed images are stored at your home directory in landsat forlder: ~/landsat
  • The image thumbnail web address that is included in the results can be used to make sure that clouds are not obscuring the subject of interest. Run the search again if you need to narrow down your result and then start downloading images. Each image is usually more than 700mb and it might takes a very long time if there are too many images to download
  • Image processing is a very heavy and resource consuming task. Each process takes about 5-10 mins. We recommend that you run the processes in smaller badges. Pansharpening, while increasing image resolution 2x, substantially increases processing time.
  • Landsat-util requires at least 2GB of Memory (RAM).
  • Make sure to read over the section on returned products as it is different depending on scene acquisition date.