Skip to main content

Archiving and Compression

Database Backup Documentation​

Archiving and Compression​

Tools Used:​

  • tar: Used for archiving files and directories.
  • pigz: A parallel implementation of gzip, used for compressing tar archives.

Backup Process:​

  1. Archiving:

    • We use tar to create an archive of database backup files and directories. Tar archives help organize and bundle multiple files into a single file for efficient storage and transportation.

  2. Compression:

    • After archiving, we use pigz to compress the tar archive. Pigz provides parallel compression, utilizing multiple CPU cores for faster compression speed.

Benefits:​

  • Efficiency: Using tar and pigz together allows us to efficiently archive and compress backup data, reducing storage space and minimizing transfer times.
  • Parallel Compression: pigz takes advantage of multiple CPU cores for faster compression, speeding up the backup process.
  • Ease of Use: Both tar and pigz are widely used and well-supported tools, making it easy to integrate them into our backup workflow.

S3 Lifecycle Policy​

Monthly Backups Lifecycle Rule:​

  • Day 0: Objects are uploaded.
  • Day 30: Objects are transitioned to Standard-IA storage class.
  • Day 90: Objects are moved to Glacier Flexible Retrieval (formerly Glacier).
  • Day 2555: Objects expire.

Weekly Backups Lifecycle Rule:​

  • Day 0: Objects are uploaded.
  • Day 60: Objects expire.

Daily Backups Lifecycle Rule:​

  • Day 0: Objects are uploaded.
  • Day 30: Objects expire.