Large processing jobs over entire archives of satellite data need specific processing power. VITO has a Hadoop processing environment with 3000+ cores and is readily available for upscaling larger datasets and to make large processing jobs more efficient. Datasets are mounted on all processing nodes for your parallel processing needs.
Product Description
This service gives access to a virtual research environment with an expansive Earth Observation data archive. After a prototyping phase on a user VM, the Hadoop processing environment is readily available for upscaling larger datasets and to make large processing jobs more efficient. Prototyping on smaller datasets can be done using a virtual machine and can be upscaled using a couple of Spark code adaptations to the Hadoop cluster. Jobs on the Hadoop platform can be monitored using default yarn tools or a custom job-control application.
The amount of Hadoop cores (CPUs) is related to the available RAM, each Hadoop Core can use up to 3.5 GB RAM. If your processing is memory intensive, estimate the memory which is needed for your processing workflow and select the amount of cores accordingly. The hours in the hourly package are CPU hours when using the full capacity of the selected Hadoop Cores.
By selecting eg. 40 CPU cores, combined with the 100 hourly package, your processing has a maximal capacity of 40 CPU cores and up to 140GB of RAM. And this for 4000 (40 x 100) CPU hours of processing. If the process needs full capacity, it can process during 100 hours. However if it needs only 1 core, it can run for a total amount of 4000 processing hours.
The total amount of CPU hours will be available for your processing services during one calendar year after request. After this period of one year, the remaining CPU hours will be expired and no longer useable for your processing service.
To see all complementing products, please click here.
Provider:
Service setup time
Service availability
Helpdesk response time
PROBA-V MEP Web Interface services overview:
- Product Catalogue Service:
REST API supporting searching EO products in the catalogue.
https://proba-v-mep.esa.int/api/catalog/ - Time series query service:
REST API supporting interactive retrieval of time series.
https://proba-v-mep.esa.int/api/timeseries/apidocs/ - N-daily compositor service:
REST API supporting processing on demand to be used in processing chains.
https://proba-v-mep.esa.int/pod-rest/ - OpenSearch:
OGC standard based web service supporting discovery of available EO data collections and products in the VITO Product Distribution Facility (PDF).
http://www.vito-eodata.be/openSearch/description.xml - WMTS:
OGC standard based web service supporting access to pre-rendered georeferenced map tiles.
https://proba-v-mep.esa.int/applications/geo-viewer/app/ - OGC WCS:
OGC Web Coverage Service for downloading of georeferenced rasters with selection of bands, time and bounding box.
https://proba-v-mep.esa.int/applications/geo-viewer/app/ - More details here:
http://proba-v-mep.esa.int/documentation/manuals/web-interfaces
EO Mission | Data Set Type | Archive Format | Availability Since | Geographic Coverage | Temporal Coverage |
---|---|---|---|---|---|
Sentinel-1 | SLC/GRD/GRD | SAFE/GEOTIFF | 2014 | BE | 0.50% |
Sentinel-2 | L2A/NDVI/FAPAR/LAI/FCOVER | GEOTIFF | 2015 | EU | 10% |
Sentinel-3 | TBC | TBC | 2016 | 100%, Vegetation only | 0% |
PROBA-V | L1C/L2A | HDF5 and GEOTIFF | 2013 | 100% | 100% |
PROBA-V | TOA/TOC/NDVI daily/5daily/10daily | HDF5 and GEOTIFF | 2013 | 100% | 100% |
SPOT-VGT | L2/daily/10daily | HDF4 | 1998-2014 | 100% | 100% |
Copernicus Global Land Service | 10daily FAPAR/LAI | NETCDF | 1998 | 100% | 100% |
Please note
Each Hadoop core comes with up to 3.5 GB RAM.
Microsoft Windows OS is available upon request.
The GPU graphics card is the NVIDIA Quadro M5000.
If you request ESA Sponsorship for this service, please allow a few extra business days for delivery.
If you aren't finding what you are looking for, please contact the Helpdesk for more information.
Pricing / Licensing
VITO and cloudeo Terms & Conditions apply.