AsyncS3Bucket#

class lsst.ts.salobj.AsyncS3Bucket(name, *, create=False, profile=None, domock=False)#

Bases: object

Asynchronous interface to an Amazon Web Services s3 bucket.

Parameters:

  • name (str) – Name of bucket. If using Amazon Web Services see <https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html> for details. In particular note that bucket names must be globally unique across all AWS accounts.

  • create (bool, optional) – If true and the bucket does not exist, create it. If false then assume the bucket exists. You will typically want true if using a mock server (domock true).

  • profile (str, optional) – Profile name; use the default profile if None.

  • domock (bool, optional) – If true then start a mock S3 server. This is recommended for running in simulation mode.

service_resource#

The resource used to access the S3 service. Primarly provided for unit tests.

Type:

boto3.resources.factory.s3.ServiceResource

name#

The bucket name.

Type:

str

profile#

The profile name, or None if not specified.

Type:

str

bucket#

The S3 bucket.

Type:

boto3.resources.s3.Bucket

Notes

Reads the following Environment Variables; follow the link for details:

  • S3_ENDPOINT_URL: The endpoint URL, e.g. http://foo.bar:9000.

The format for bucket names, file keys, and largeFileEvent event URLs is described in CAP 452

Methods Summary

download(key[, callback])

Download a file-like object from the bucket.

exists(key)

Check if a specified file exists in the bucket.

make_bucket_name(s3instance[, s3category])

Make an S3 bucket name.

make_key(salname, salindexname, generator, date)

Make a key for an item of data.

size(key)

Get the size in bytes of a given file in the bucket.

stop_mock()

Stop the mock s3 service, if running.

upload(fileobj, key[, callback])

Upload a file-like object to the bucket.

Methods Documentation

async download(key, callback=None)#

Download a file-like object from the bucket.

Parameters:

  • key (str) – Name of the file in the bucket.

  • callback (callable, optional) – Function to call with updates while writing. The function receives one argument: the number of bytes read so far. If the transfer is successful then it will always be called at least once, and the sum of the number of bytes for all calls will equal the size of the file.

Returns:

fileobj – The downloaded data as a file-like object.

Return type:

io.BytesIO

Notes

To convert a file-like object fileobj to an astropy.io.fits.HDUList named hdulist:

hdulist = astropy.io.fits.open(fileobj)

The callback function is called by S3.Bucket.download_fileobj.

async exists(key)#

Check if a specified file exists in the bucket.

Parameters:

key (str) – Name of the potential file in the bucket.

Return type:

bool

static make_bucket_name(s3instance, s3category='LFA')#

Make an S3 bucket name.

Parameters:

  • s3instance (str) – S3 server instance. Typically “Summit”, “Tucson” or “NCSA”.

  • s3category (str, optional) – Category of S3 server. The default is “LFA”, for the Large File Annex.

Returns:

bucket_name – The S3 bucket name in the format described below:

Return type:

str

Raises:

ValueError – If one or more arguments does not meet the rules below or the resulting bucket name is longer than 63 characters.

Notes

The rules for all arguments are as follows:

  • Each argument must start and end with a letter or digit.

  • Each argument may only contain letters, digits, and “.”.

The returned bucket name is cast to lowercase (because S3 bucket names may not contain uppercase letters) and has format:

rubinobs-{s3category}-{s3instance}]
static make_key(salname, salindexname, generator, date, other=None, stem=None, suffix='.dat')#

Make a key for an item of data.

Parameters:

  • salname (str) – SAL component name, e.g. ATPtg.

  • salindexname (str, int, or None) – For an indexed SAL component: a name associated with the SAL index, or just the index itself if there is no name. Specify None for a non-indexed SAL component. For example: “MT” for Main Telescope, “AT” for auxiliary telescope, or “ATBlue” for the AT Blue Fiber Spectrograph.

  • generator (str) – Dataset type.

  • date (astropy.time.Time) – A date – typically the date the data was taken.

  • other (str or None, optional) – Additional text to identify the data and make the key unique. If None use date.tai.isot: date as TAI, in ISO-8601 format, with a “T” between the date and time and a precision of milliseconds.

  • stem (str, optional) – Use this parameter for the full file stem in place of the genrated name.

  • suffix (str, optional) – Key suffix, e.g. “.fits”.

Returns:

key – The key, as described below.

Return type:

str

Notes

The returned key has format:

{fullsalname}/{generator}/{yyyy}/{mm}/{dd}/
    {fullsalname}-{generator}-{other}{suffix}

where:

  • fullsalname = {salname}:{salindexname} if salindexname, else salname.

  • yyyy, mm, dd are the “observing day”: the year, month and day at TAI date - 12 hours, with 4, 2, 2 digits, respectively. The “observing day” does not change during nighttime observing at the summit. Year, month and day are determined after rounding the date to milliseconds, so the reported observing day is consistent with the default value for other.

If the stem option is used, the returned key has format:

{fullsalname}/{generator}/{yyyy}/{mm}/{dd}/{stem}{suffix}

Note that the url field of the largeFileObjectAvailable event should have the format f”s3://{bucket}/{key}”

async size(key)#

Get the size in bytes of a given file in the bucket.

Parameters:

key (str) – Name of the file in the bucket.

Return type:

int

stop_mock()#

Stop the mock s3 service, if running. A no-op if not running.

Return type:

None

async upload(fileobj, key, callback=None)#

Upload a file-like object to the bucket.

Parameters:

  • fileobj (file-like object) – File-like object that can be read as binary data.

  • key (str) – Name to use for the file in the bucket.

  • callback (callable, optional) – Synchronous function to call with updates while writing. The function receives one argument: the number of bytes written. If the transfer is successful then it will always be called at least once, and the sum of the number of bytes for all calls will equal the size of the file. The function is called by the boto3 library, which is why it must be synchronous.

Returns:

url – The assembled URL needed for a largeFileObjectAvailable event URL.

Return type:

str

Notes

To create a file-like object fileobj from an astropy.io.fits.HDUList named hdulist:

fileobj = io.BytesIO()
hdulist.writeto(fileobj)
fileobj.seek(0)

The returned URL has the format:

{S3_ENDPOINT_URL}/{bucket_name}/{key}