Ordering Sentinel-2 products from Long Term Archive with sen2r

October 21, 2019
By

[This article was first published on R on Luigi Ranghetti Website, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Until August 2019, all
Sentinel-2 satellite data
could be directly downloaded from the ESA Data Hub, both through the interactive
Open Hub or using an
API interface.

Recently this policy was changed: typically, only the most recent products
are available for direct download, while the oldest ones (level 2A archives
older than 18 months and level 1C older than one year) are stored in the so called
Long Term Archive,
and must be ordered by the user; then, they are made available
for download after a while (no messages are sent to the user).
For further details, see the
announcement of activation of LTA access for Sentinel-2 and 3.

R users can exploit a functionality recent implemented in the package
sen2r
to deal with products no more available for direct download.

Overview and installation

sen2r
is a package devoted to download and preprocess Sentinel-2 satellite imagery
via an accessible and easy to install interface, which can also be used to
manage massive processing operations and to schedule automatic processing chains.
For an overview of the packages functionalities, see this previous post.

sen2r
was recently released on CRAN,
but the current CRAN release (1.1.0) does not yet support ordering products from the
Long Term Archive (LTA);
to exploit this functionality, it is necessary to use the GitHub release
(version 1.2.0 or higher), which can be installed using the package remotes:

remotes::install_github("ranghetti/sen2r")

(refer to this page
for a detailed installation guide, including the installation of system dependences).

After installing the package, the following function should be used to save the
SciHub credentials:

library(sen2r)
write_scihub_login("my_user", "my_password")

Automatic orders

sen2r
can be used both in online and offline mode: in the first case, Sentinel-2 source
archives (in SAFE format) are searched and downloaded from ESA SciHub, while in
the second case, only archives already present on the user machine are used.
This post refers to the online usage of the package (nothing changes using it
in offline mode).

Starting from version 1.2.0,
SAFE archives which are not available for direct download are automatically ordered:
this means that, in the case some archives – required to produce the users’ outputs –
are stored in the LTA:

  1. they are ordered;
  2. the processing chain launched by the user continues skipping these archives
    (and so without producing the consequent output rasters);
  3. the user can check the order status with a specific command.

Let’s see some examples of that.

Note: in the following examples, we will search / download an area of interest
located in a random Sentinel-2 tile of Kazakhstan, in a time window of 20 days.
Randomness is necessary tyo catch products not available for download
(otherwise, archives overlapping a fixed area of interest would be ordered
the first time I ran examples, so making them available and compromising the subsequent
executions).
This expedient does not ensure reproducibility, since the randomly selected products
could already be available for different reasons.

Using the function sen2r()

sen2r()
is the main function of the package, which can be used to manage a whole
processing chain.
In the following example sen2r() will be used to launch a processing chain
to generate RGB colour images from level 1C (top of atmosphere reflectances) data.
First of all, the boundaries of Kazakhstan are downloaded and used to select
the Sentinel-2 tiles intersecting this country.
Then, a specific area of interest (the centroid of a random tile, with a buffer
of 5 km) is created.

library(sf); library(magrittr)
download.file(
  "https://biogeo.ucdavis.edu/data/gadm3.6/Rsf/gadm36_KAZ_0_sf.rds",
  kaz_sf_path <- file.path(tempdir(), "gadm36_KAZ_0_sf.rds")
)
kaz_sf <- readRDS(kaz_sf_path)

s2tiles_kaz <- tiles_intersects(kaz_sf, all = TRUE, out_format = "sf")

In this example,
tiles_intersects() is a
sen2r
function which allows obtaining the Sentinel-2 tiles which cover a specific
area of interest.
Then, the processing is launched:

safe_folder <- tempfile(pattern = "safe_")
out_folder_1 <- tempfile(pattern = "Example1_")
sel_tile_1 <- s2tiles_kaz[sample(nrow(s2tiles_kaz),1),] %>%
  sf::st_transform(3857)
sel_tile_1$tile_id
## [1] "40TET"
sel_extent_1 <- sf::st_centroid(sel_tile_1) %>% st_buffer(5e3)
out1 <- sen2r(
  gui = FALSE,
  extent = sel_extent_1,
  timewindow = c("2018-02-21", "2018-03-02"),
  list_rgb = "RGB432T",
  path_l1c = safe_folder,
  path_l2a = safe_folder,
  path_out = out_folder_1,
  log = log_path_1 <- tempfile()
)

This is the log of the function (by default it is returned at standard output,
while in this example it was redirected to a temporary file by setting
the argument log):

## [2019-10-24 15:27:02] Starting sen2r execution.
## [2019-10-24 15:27:02] Searching for available SAFE products on SciHub...
## [2019-10-24 15:27:09] Ordering 1 Sentinel-2 images stored in the Long Term Archive...
## [2019-10-24 15:27:09] 1 of 1 Sentinel-2 images were correctly ordered. You can check at a later time if the ordered products were made available using the command:
## 
## safe_is_online("/home/lranghetti/.sen2r/lta_orders/lta_20191024_152709.json")
## 
## [2019-10-24 15:27:09] Computing output names...
## [2019-10-24 15:27:10] Starting to download the required level-2A SAFE products.
## [2019-10-24 15:27:10] Download of level-2A SAFE products terminated.
## [2019-10-24 15:27:10] Starting to download the required level-1C SAFE products.
## [2019-10-24 15:27:10] Check if products are available for download...
## [2019-10-24 15:27:11] 1 Sentinel-2 images are already available and will not be ordered.
## [2019-10-24 15:27:11] Downloading Sentinel-2 image 1 of 1 (S2B_MSIL1C_20180301T070819_N0206_R106_T40TET_20180301T105657.SAFE)...
## [2019-10-24 15:28:45] Download of level-1C SAFE products terminated.
## [2019-10-24 15:28:45] Updating output names...
## [2019-10-24 15:28:45] Starting to translate SAFE products in custom format.
## GDAL version in use: 2.2.3
## Using UTM zone 40.
## 1 output files were correctly created.
## [2019-10-24 15:28:45] Starting to merge tiles by orbit.
## Using projection "+proj=utm +zone=40 +datum=WGS84 +units=m +no_defs".
## [2019-10-24 15:28:46] Starting to edit geometry (clip, reproject, rescale).
## [2019-10-24 15:28:46] Producing required RGB images.
## 1 output RGB files were correctly created.
## [2019-10-24 15:28:50] Generating thumbnails.
## 1 output files were correctly created.
## [2019-10-24 15:28:51] Execution of sen2r session terminated.

The required archive, not available for direct download, were ordered.
After doing that, function continued processing the available archives.

At the end of the processing, a warning could appear in case the user exceeded the order quota:

##  of  Sentinel-2 images were not correctly ordered because user '' offline products retrieval quota exceeded. Please retry later, otherwise use different SciHub credentials (see ?write_scihub_login or set a specific value for argument "apihub"). 

The function cited in the log can be used to check if the order was processed:

## safe_is_online("/home/lranghetti/.sen2r/lta_orders/lta_20191024_152709.json")
## S2A_MSIL1C_20180224T070851_N0206_R106_T40TET_20180224T110454.SAFE 
##                                                             FALSE

(the JSON file, automatically created by sen2r(), contains the URLs
of the ordered products).

Whenever this function will be returning TRUE for all the listed archives,
the previous sen2r() execution can be re-launched to complete the processing.

In the case the user does not want to order missing products, it is sufficient
to set the sen2r() argument order_lta to FALSE.
Launching the function with the Graphical User Interface, this setting can be set
in the first sheet (see the screenshot below).

Using specific functions

Other package functions can be used to perform specific steps of a processing chain:

  • s2_list() allows searching
    the SAFE archives matching the required parameters.
    By default, all archives (downloadable / stored in LTA) are returned;
    setting the new argument availability to "online" or "lta" allows
    returning only available archives or LTA products, respectively.
    Setting availability = "check" allows distinguish which returned products
    are available for download and which are not:

    out_folder_2 <- tempfile(pattern = "Example2_")
    sel_tile_2 <- s2tiles_kaz[sample(nrow(s2tiles_kaz),1),]
    sel_tile_2$tile_id
    ## [1] "42UXF"
    out_list_2 <- s2_list(
      tile = sel_tile_2$tile_id, 
      time_interval = c("2018-02-21", "2018-03-02"),
      availability = "check"
    )
    # Show available products
    out_list_2[attr(out_list_2, "online")]
    ##                                      S2A_MSIL1C_20180301T061801_N0206_R034_T42UXF_20180301T082121.SAFE 
    ## "https://scihub.copernicus.eu/apihub/odata/v1/Products('ef9d324a-1172-42f5-8396-b120ab4dab89')/$value" 
    ##                                      S2B_MSIL1C_20180302T063759_N0206_R120_T42UXF_20180302T102521.SAFE 
    ## "https://scihub.copernicus.eu/apihub/odata/v1/Products('9efefd9f-563c-4ab4-bdb1-e7178688c551')/$value"
    # Show products stored in LTA
    out_list_2[attr(out_list_2, "lta")]
    ##                                      S2A_MSIL1C_20180222T062851_N0206_R077_T42UXF_20180222T101014.SAFE 
    ## "https://scihub.copernicus.eu/apihub/odata/v1/Products('e6f94a43-804c-4fa7-a216-8ffc6620b8ed')/$value" 
    ##                                      S2B_MSIL1C_20180224T061829_N0206_R034_T42UXF_20180224T101548.SAFE 
    ## "https://scihub.copernicus.eu/apihub/odata/v1/Products('7766d2f8-da66-427f-8e19-b134c4c482f3')/$value" 
    ##                                      S2A_MSIL1C_20180225T063831_N0206_R120_T42UXF_20180225T103712.SAFE 
    ## "https://scihub.copernicus.eu/apihub/odata/v1/Products('bff08819-ce0e-44b5-8327-4e379fc7542c')/$value" 
    ##                                      S2B_MSIL1C_20180227T062809_N0206_R077_T42UXF_20180227T083242.SAFE 
    ## "https://scihub.copernicus.eu/apihub/odata/v1/Products('d44b1929-5b77-47a1-865c-7515dcab9960')/$value"
  • s2_download()
    can be used to download specific SAFE products.
    By default, products stored in LTA are automatically ordered (unless
    argument order_lta is set to FALSE by the user):

    s2_download(out_list_2, order_lta = FALSE)

    This function can also be used passing it the path of a JSON file
    created by sen2r() (see above) or by s2_order() (see below)
    containing the URLs of the archives to be downloaded.

  • safe_is_online()
    function, which was already shown, can also be used with a list of
    required archives:

    safe_is_online(out_list_2)
    ## S2A_MSIL1C_20180222T062851_N0206_R077_T42UXF_20180222T101014.SAFE 
    ##                                                             FALSE 
    ## S2B_MSIL1C_20180224T061829_N0206_R034_T42UXF_20180224T101548.SAFE 
    ##                                                             FALSE 
    ## S2A_MSIL1C_20180225T063831_N0206_R120_T42UXF_20180225T103712.SAFE 
    ##                                                             FALSE 
    ## S2B_MSIL1C_20180227T062809_N0206_R077_T42UXF_20180227T083242.SAFE 
    ##                                                             FALSE 
    ## S2A_MSIL1C_20180301T061801_N0206_R034_T42UXF_20180301T082121.SAFE 
    ##                                                              TRUE 
    ## S2B_MSIL1C_20180302T063759_N0206_R120_T42UXF_20180302T102521.SAFE 
    ##                                                              TRUE

Manual orders

The new function s2_order()
can be used to manually order SAFE archives stored on LTA.
Similarly to s2_download() and safe_is_online(), s2_order() accepts a list
of SAFE URLs to be downloaded, both as R vector of as path of a JSON file
containing them:

ordered_list <- s2_order(out_list_2)
## [2019-10-24 15:29:19] Check if products are already available for download...
## [2019-10-24 15:29:22] 2 Sentinel-2 images are already available and will not be ordered.
## [2019-10-24 15:29:22] Ordering 4 Sentinel-2 images stored in the Long Term Archive...
## [2019-10-24 15:29:38] 4 of 4 Sentinel-2 images were correctly ordered. You can check at a later time if the ordered products were made available using the command:
## 
## safe_is_online("/home/lranghetti/.sen2r/lta_orders/lta_20191024_152938.json")
ordered_list
##                                      S2A_MSIL1C_20180222T062851_N0206_R077_T42UXF_20180222T101014.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('e6f94a43-804c-4fa7-a216-8ffc6620b8ed')/$value" 
##                                      S2B_MSIL1C_20180224T061829_N0206_R034_T42UXF_20180224T101548.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('7766d2f8-da66-427f-8e19-b134c4c482f3')/$value" 
##                                      S2A_MSIL1C_20180225T063831_N0206_R120_T42UXF_20180225T103712.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('bff08819-ce0e-44b5-8327-4e379fc7542c')/$value" 
##                                      S2B_MSIL1C_20180227T062809_N0206_R077_T42UXF_20180227T083242.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('d44b1929-5b77-47a1-865c-7515dcab9960')/$value" 
## attr(,"available")
##                                      S2A_MSIL1C_20180301T061801_N0206_R034_T42UXF_20180301T082121.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('ef9d324a-1172-42f5-8396-b120ab4dab89')/$value" 
##                                      S2B_MSIL1C_20180302T063759_N0206_R120_T42UXF_20180302T102521.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('9efefd9f-563c-4ab4-bdb1-e7178688c551')/$value" 
## attr(,"notordered")
## named character(0)
## attr(,"path")
## [1] "/home/lranghetti/.sen2r/lta_orders/lta_20191024_152938.json"

The function returns a vector of ordered products.
Some attributes are eventually added:

  • "available" contains products not ordered because already available for download;
  • "notordered" contains products whose orders failed (commonly because the
    user exceeded his quota);
  • "path" contains the path of the saved JSON file containing the URLs of
    the ordered products, which can be used with functions safe_is_online()
    and s2_download() to check if the order was processed and then to download
    products (the creation of this file can be skipped setting
    export_prodlist = FALSE).

Conclusions

Starting from version 1.2.0,
sen2r
is able to manage Sentinel-2 products stored in the Long Term Archive (LTA)
and so not directly downloadable.
The default behaviour of package functions is to order unavailable products,
processing only available archives and providing the way to check the order status
through a single line of code.
Once the order was processed, the user can relaunch the same code to complete
the processing.
New functions can be exploited to perform specific steps:
safe_is_online()
can be used to check if SAFE archives are / were made available, and
s2_order() to manually
order them.
The implementation of additional features related to LTA is planned for the
future (e.g. an additional HTML report with the list of used / ordered
products).

These features were implemented recently, so users could encounter some issues
exploiting them.
With the exception of issues not depending on this package (e.g. errors due to
invalid SciHub credentials, or if user quota exceeded), users are encouraged to
report them on GitHub.

Credits


sen2r
is developed by Luigi Ranghetti and Lorenzo Busetto
(IREA-CNR),
and it is released under the GNU GPL-3 license.

Using sen2r
for production (including scientific products) requires to cite it
(use this entry).

To leave a comment for the author, please follow the link and comment on their blog: R on Luigi Ranghetti Website.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.



If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.

Search R-bloggers

Sponsors

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)