avroconvert.sources.s3 package

Submodules

avroconvert.sources.s3.reader module

class avroconvert.sources.s3.reader.S3(access_key: Optional[str] = None, secret_key: Optional[str] = None, session_token: Optional[str] = None, bucket: Optional[str] = None, prefix: str = '', datatype: str = 'avro')[source]

Bases: object

A class used to read files from amazon s3 and convert them to bytes

Parameters

  • access_key (str) – AWS access key id

  • secret_key (str) – AWS secret access key

  • session_token (str) – AWS session token (if any)

  • bucket (str) – Name of the bucket in s3. This is where the avro file is read from

  • datatype (str) – format of the source file, which will be read from s3. Default value is avro

  • prefix (str) – prefix is the starting letters of the file names in the cloud storage. For example, if the bucket contains files with name test-01, test-02 and test-03, the file prefix can be test. All the files with this prefix will be read

get_data() → dict[source]

Lists all files in S3 (filtered by prefix, if it is passed), reads the files as bytes and returns a list of data read from the files

Returns

list all files’ data from s3. Each element of the list is bytes

Return type

list

Module contents