avroconvert.sources.filesystem package¶

Submodules¶

class avroconvert.sources.filesystem.reader.FileSystem(bucket: str, prefix: Optional[str] = None, datatype: str = 'avro')[source]¶

Bases: object

Parameters

datatype –

str: format of the source file, which will be read from google cloud storage. Default value is avro

type datatype: str
param prefix: str prefix is the starting letters of the file names in the cloud storage. For example, if the bucket contains files with name test-01, test-02 and test-03, the file prefix can be test. All the files with this prefix will be read
type prefix: str

Lists all files in the local folder (filtering by prefix if one is provided), reads them as bytes, and returns a list of data read from the files.

Returns: list of bytes where each element of the list represents a file read (in bytes) from local folders.
Return type: list

read_files(filename: str)[source]¶

Read file from local filesystem and convert it into bytes

Parameters: filename (str) – Name of the file to read from local folder
Returns: avro file from local folder, converted to bytes
Return type: bytes