AsyncS3Bucket

class lsst.ts.salobj.AsyncS3Bucket(name: str, *, create: bool = False, profile: Optional[str] = None, domock: bool = False)

Bases: object

Asynchronous interface to an Amazon Web Services s3 bucket.

Parameters
namestr

Name of bucket. If using Amazon Web Services see <https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html> for details. In particular note that bucket names must be globally unique across all AWS accounts.

createbool, optional

If True and the bucket does not exist, create it. If False then assume the bucket exists. You will typically want true if using a mock server (domock true).

profilestr, optional

Profile name; use the default profile if None.

domockbool, optional

If True then start a mock S3 server. This is recommended for running in simulation mode.

Notes

Reads the following Environment Variables; follow the link for details:

  • S3_ENDPOINT_URL: The endpoint URL, e.g. http://foo.bar:9000.

The format for bucket names, file keys, and largeFileEvent event URLs is described in CAP 452

Attributes
service_resourceboto3.resources.factory.s3.ServiceResource

The resource used to access the S3 service. Primarly provided for unit tests.

namestr

The bucket name.

profilestr

The profile name, or None if not specified.

bucketboto3.resources.s3.Bucket

The S3 bucket.

Methods Summary

download(key[, callback])

Download a file-like object from the bucket.

exists(key)

Check if a specified file exists in the bucket.

make_bucket_name(s3instance[, s3category])

Make an S3 bucket name.

make_key(salname, salindexname, generator, date)

Make a key for an item of data.

size(key)

Get the size in bytes of a given file in the bucket.

stop_mock()

Stop the mock s3 service, if running.

upload(fileobj, key[, callback])

Upload a file-like object to the bucket.

Methods Documentation

async download(key: str, callback: Optional[Callable[[int], None]] = None) _io.BytesIO

Download a file-like object from the bucket.

Parameters
keystr

Name of the file in the bucket.

callbackcallable, optional

Function to call with updates while writing. The function receives one argument: the number of bytes read so far. If the transfer is successful then it will always be called at least once, and the sum of the number of bytes for all calls will equal the size of the file.

Returns
fileobjio.BytesIO

The downloaded data as a file-like object.

Notes

To convert a file-like object fileobj to an astropy.io.fits.HDUList named hdulist:

hdulist = astropy.io.fits.open(fileobj)

The callback function is called by S3.Bucket.download_fileobj.

async exists(key: str) bool

Check if a specified file exists in the bucket.

Parameters
keystr

Name of the potential file in the bucket.

static make_bucket_name(s3instance: str, s3category: str = 'LFA') str

Make an S3 bucket name.

Parameters
s3instancestr

S3 server instance. Typically “Summit”, “Tucson” or “NCSA”.

s3categorystr, optional

Category of S3 server. The default is “LFA”, for the Large File Annex.

Returns
bucket_namestr

The S3 bucket name in the format described below:

Raises
ValueError

If one or more arguments does not meet the rules below or the resulting bucket name is longer than 63 characters.

Notes

The rules for all arguments are as follows:

  • Each argument must start and end with a letter or digit.

  • Each argument may only contain letters, digits, and “.”.

The returned bucket name is cast to lowercase (because S3 bucket names may not contain uppercase letters) and has format:

rubinobs-{s3category}-{s3instance}]
static make_key(salname: str, salindexname: Optional[Union[str, int]], generator: str, date: astropy.time.Time, other: Optional[str] = None, suffix: str = '.dat') str

Make a key for an item of data.

Parameters
salnamestr

SAL component name, e.g. ATPtg.

salindexnamestr, int, or None

For an indexed SAL component: a name associated with the SAL index, or just the index itself if there is no name. Specify None for a non-indexed SAL component. For example: “MT” for Main Telescope, “AT” for auxiliary telescope, or “ATBlue” for the AT Blue Fiber Spectrograph.

generatorstr

Dataset type.

dateastropy.time.Time

A date – typically the date the data was taken.

otherstr or None, optional

Additional text to identify the data and make the key unique. If None use date.tai.isot: date as TAI, in ISO-8601 format, with a “T” between the date and time and a precision of milliseconds.

suffixstr, optional

Key suffix, e.g. “.fits”.

Returns
keystr

The key, as described below.

Notes

The returned key has format:

{fullsalname}/{generator}/{yyyy}/{mm}/{dd}/
    {fullsalname}-{generator}-{other}{suffix}

where:

  • fullsalname = {salname}:{salindexname} if salindexname, else salname.

  • yyyy, mm, dd are the “observing day”: the year, month and day at TAI date - 12 hours, with 4, 2, 2 digits, respectively. The “observing day” does change during nighttime observing at the summit. Year, month and day are determined after rounding the date to milliseconds, so the reported observing day is consistent with the default value for other.

Note that the url field of the largeFileObjectAvailable event should have the format f”s3://{bucket}/{key}”

async size(key: str) int

Get the size in bytes of a given file in the bucket.

Parameters
keystr

Name of the file in the bucket.

stop_mock() None

Stop the mock s3 service, if running. A no-op if not running.

async upload(fileobj: BinaryIO, key: str, callback: Optional[Callable[[int], None]] = None) None

Upload a file-like object to the bucket.

Parameters
fileobjfile-like object

File-like object that can be read as binary data.

keystr

Name to use for the file in the bucket.

callbackcallable, optional

Function to call with updates while writing. The function receives one argument: the number of bytes written. If the transfer is successful then it will always be called at least once, and the sum of the number of bytes for all calls will equal the size of the file. The callback function is called by S3.Bucket.upload_fileobj.

Notes

To create a file-like object fileobj from an astropy.io.fits.HDUList named hdulist:

fileobj = io.BytesIO()
hdulist.writeto(fileobj)
fileobj.seek(0)