avroconvert.sources.gcs package

Submodules

avroconvert.sources.gcs.reader module

class avroconvert.sources.gcs.reader.GCS(auth_file: Optional[str] = None, bucket: Optional[str] = None, datatype: str = 'avro', prefix: Optional[str] = None)[source]

Bases: object

A class used to read files from google cloud storage and convert them to list of bytes

Parameters

  • client (gcs bucket client) – google cloud storage bucket client object. With this, we can interact with google cloud’s bucket

  • bucket (str) – Name of the bucket in google cloud storage. This is where the avro file is read from

  • datatype (str) – format of the source file, which will be read from google cloud storage. Default value is avro

  • prefix (str) – str prefix is the starting letters of the file names in the cloud storage. For example, if the bucket contains files with name test-01, test-02 and test-03, the file prefix can be test. All the files with this prefix will be read

get_data() → list[source]

Lists all files in S3 (filtering by prefix if one is provided), reads them as bytes, and returns a list of data read from the files.

Returns

list of bytes where each element of the list represents a file read (in bytes) from Google Storage Bucket.

Return type

list

Module contents