AsyncS3Bucket¶

class lsst.ts.salobj.AsyncS3Bucket(name, domock=False)¶

Bases: object

Asynchronous interface to an Amazon Web Services s3 bucket.

Parameters:	name : `str` Name of bucket. If using Amazon Web Services see <https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html> for details. In particular note that bucket names must be globally unique across all AWS accounts. domock : `bool` If True then start a mock S3 server. This is recommended for running in simulation mode.

Notes

Reads the following Environment Variables; follow the link for details:

S3_ENDPOINT_URL: The endpoint URL, e.g. http://foo.bar:9000.

The format for bucket names, file keys, and largeFileEvent event URLs is described in CAP 452

Attributes:	service_resource : `boto3.resources.factory.s3.ServiceResource` The resource used to access the S3 service. Primarly provided for unit tests. name : `str` The bucket name bucket : `boto3.resources.s3.Bucket` The S3 bucket.

Methods Summary

`download`(key[, callback])	Download a file-like object from the bucket.
`exists`(key)	Check if a specified file exists in the bucket.
`make_bucket_name`(salname, salindexname, …)	Make an S3 bucket name.
`make_key`(salname, generator, date)	Make a key for an item of data.
`size`(key)	Get the size in bytes of a given file in the bucket.
`stop_mock`()	Stop the mock s3 service, if running.
`upload`(fileobj, key[, callback])	Upload a file-like object to the bucket.

Methods Documentation

download(key, callback=None)¶

Download a file-like object from the bucket.

Parameters:	key : `str` Name of the file in the bucket. callback : callable (optional) Function to call with updates while writing. The function receives one argument: the number of bytes read so far. If the transfer is successful then it will always be called at least once, and the sum of the number of bytes for all calls will equal the size of the file.
Returns:	fileobj : `io.BytesIO` The downloaded data as a file-like object.

Notes

To convert a file-like object fileobj to an astropy.io.fits.HDUList named hdulist:

hdulist = astropy.io.fits.open(fileobj)

The callback function is called by S3.Bucket.download_fileobj.

exists(key)¶

Check if a specified file exists in the bucket.

Parameters:	key : `str` Name of the potential file in the bucket.

static make_bucket_name(salname, salindexname, s3instance, s3category='LFA')¶

Make an S3 bucket name.

Parameters:	salname : `str` SAL component name, e.g. ATPtg. salindexname : `str` or `None` For an indexed SAL component: a name associated with the SAL index, or just the index itself if there is no name. Specify `None` for a non-indexed SAL component. For example: “MT” for Main Telescope, “AT” for auxiliary telescope, or “ATBlue” for the AT Blue Fiber Spectrograph. s3instance : `str` S3 server instance. Typically “Summit”, “Tucson” or “NCSA”. s3category : `str` (optional) Category of S3 server. The default is “LFA”, for the Large File Annex.
Returns:	bucket_name : `str` The S3 bucket name in the format described below:
Raises:	ValueError If one or more arguments does not meet the rules below or the resulting bucket name is longer than 63 characters.

Notes

The rules for all arguments are as follows:

Each argument must start and end with a letter or digit.
Each argument may only contain letters, digits, and “.”.

The returned bucket name is cast to lowercase (because S3 bucket names may not contain uppercase letters) and has format:

rubinobs-{s3category}-{s3instance}-{salname}[.{salindexname}]

static make_key(salname, generator, date)¶

Make a key for an item of data.

Parameters:	salname : `str` SAL component name, e.g. ATPtg. generator : `str` Dataset type. date : `astropy.time.Time` Date for the key.
Returns:	key : `str` The key, as described below.

Notes

The returned key has format:

{yyyy}/{mm}/{dd}/{salname}-{generator}-{yyyy}{mm}{dd}-{tai_iso}

where:

yyyy, mm, dd are the year, month and day at date - 12 hours, with 4, 2, 2 digits, respectively. This shifted date will not change during nighttime observing at the summit.
salname is the SAL component name
generator is the dataset type, also provided as a field in the largeFileObjectAvailable event.
tai_iso is the TAI date in ISO-8601 format, with a “T” between the date and time and 3 digits of precision.

The ISO date and shifted date both use a precision of milliseconds. So if date is just before noon and rounds up to noon, the shifted date be on the same day. This is to reduce confusion by making the date and shifted date self-consistent.

Note that the url field of the largeFileObjectAvailable event should have the format f”s3://{bucket}/{key}”

size(key)¶

Get the size in bytes of a given file in the bucket.

Parameters:	key : `str` Name of the file in the bucket.

stop_mock()¶: Stop the mock s3 service, if running. A no-op if not running.

upload(fileobj, key, callback=None)¶

Upload a file-like object to the bucket.

Parameters:

fileobj : file-like object: File-like object that can be read as binary data.
key : str: Name to use for the file in the bucket.
callback : callable (optional): Function to call with updates while writing. The function receives one argument: the number of bytes written. If the transfer is successful then it will always be called at least once, and the sum of the number of bytes for all calls will equal the size of the file. The callback function is called by S3.Bucket.upload_fileobj.

Notes

To create a file-like object fileobj from an astropy.io.fits.HDUList named hdulist:

fileobj = io.BytesIO()
hdulist.writeto(fileobj)
fileobj.seek(0)

Navigation

AsyncS3Bucket¶