avroconvert package

Subpackages

• avroconvert.sources package
  • Subpackages
    • avroconvert.sources.gcs package
      • Submodules
      • avroconvert.sources.gcs.reader module
      • Module contents
    • avroconvert.sources.s3 package
      • Submodules
      • avroconvert.sources.s3.reader module
      • Module contents
  • Module contents

Submodules

avroconvert.avroconvert module

Main module.

class avroconvert.avroconvert.AvroConvert(outfolder: str, dst_format: str = 'parquet', header: bool = True)

Bases: object

A class used to read avro files and convert them to csv, parquet and json format

Parameters

• outfolder (str) – output folder to write the output files to

• header (bool) – Extracts header from the file if it is set to True

• dst_format (str) – Specifies the format to convert the avro data to

convert_avro(filename: str, data: bytes) → str

Reads byte data, converts it to avro format and writes the data to the local filesystem to the output format specified.

Parameters

• filename (str) – Name of the input file (with it’s source path). The output file will be saved by the same name, within the same folder hierarchy as it was in the source file system. The extension will be changed as per the given output format

• data (bytes) – Contains raw data in the form of bytes as read from filesystem, google cloud storage or S3. Multiple files are read sequentially and their respective data is appended to this list which is passed as the variable data

Returns

File name with path of the output file

Return type

str

avroconvert.cli module

Console script for avroconvert.

avroconvert.cli.get_config_option(config: configparser.ConfigParser, section: str, option: str)

avroconvert.cli.main()

Console script for avroconvert.

avroconvert.execute module

class avroconvert.execute.Execute(source: str, bucket: str, dst_format: str, outfolder: str, prefix: str = '', **kwargs)

Bases: object

run() → bool

Executor method for the AvroConverter class. This method parallelizes the execution for all the file read->convert->write operations.

avroconvert.log_source module

class avroconvert.log_source.Logging(log_level, filename=None)

Bases: object

get_logger()

Module contents

Top-level package for avroconvert.