Skip to main content

SeaweedFS

What it is. SeaweedFS is an S3-compatible distributed object store (chart seaweedfs/seaweedfs 4.25.1, appVersion 4.25).

Why stardelt uses it. SeaweedFS provides the default object store for the lakehouse. Iceberg tables store Parquet data files in S3-compatible storage, and SeaweedFS serves that role in the platform.

How it fits. SeaweedFS runs in the data plane and is used by Trino (via S3) and Lakekeeper as the storage backend. It is the default object store; organizations can bring their own S3-compatible storage. See Services and How it works.

What you can tune. Key settings from stardelt-platform/helm-values/seaweedfs.yaml:

SettingValues fileKeyDefaultWhat it changes
Master replicas/resourceshelm-values/seaweedfs.yamlmaster.replicas, master.resources1, requests: {cpu: "50m", memory: "128Mi"}, limits: {cpu: "1", memory: "256Mi"}Master node configuration.
Volume PVC size/slotshelm-values/seaweedfs.yamlvolume.dataDirs[0].size, volume.dataDirs[0].maxVolumes20Gi, 100Volume storage size and max volume slots.
Filer resourceshelm-values/seaweedfs.yamlfiler.resourcesrequests: {cpu: "50m", memory: "128Mi"}, limits: {cpu: "1", memory: "512Mi"}Filer resources.
S3 enabled/authhelm-values/seaweedfs.yamls3.enabled, s3.enableAuthtrue, trueEnables S3 API with authentication.
S3 port/bucketshelm-values/seaweedfs.yamls3.port, s3.createBuckets8333, [{"name": "lakehouse", "anonymousRead": false}, {"name": "backups", "anonymousRead": false}]S3 endpoint port and buckets created by the chart.
Admin UIhelm-values/seaweedfs.yamladmin.enabledtrueEnables the cluster-internal SeaweedFS admin UI.

What it replaces. SeaweedFS replaces HDFS as the default object store for this platform.

Operating it.

  • Health check: Check SeaweedFS pods in the cluster (e.g. kubectl -n stardelt get pods -l app.kubernetes.io/name=seaweedfs).
  • Logs: View logs by component (master/volume/filer/s3) with kubectl -n stardelt logs -l app.kubernetes.io/name=seaweedfs.
  • Install step: make step-storage (see stardelt-platform/Makefile) installs SeaweedFS and creates buckets.
  • Buckets: The notebooks bucket is created by stardelt-platform/scripts/create-notebooks-bucket.sh (versioned). The backup bucket is created by stardelt-platform/scripts/create-backup-bucket.sh.
  • Backup: Object store data is archived by manifests/object-store-backup.yaml (daily aws s3 sync to lakehouse-backup). See Backups.
  • Admin UI: Cluster-internal admin UI is available (see helm-values/seaweedfs.yaml); access via make seaweedfs-admin.