3.25 Releases

Convox 3.25 upgrades Kubernetes to 1.35 across all providers. The 3.25.3 release adds additional node pools to GCP Racks with GPU and single-host Cloud TPU support, adds GPU observability on GCP, produces multi-arch App images on Karpenter Racks whose architectures span both amd64 and arm64, makes the BuildArch App parameter effective, adds the cloudwatch_disable and karpenter_build_disruption_budget_nodes rack parameters for AWS, and replaces AWS additional node groups create-first so a replacing configuration change creates the new group before removing the old one. The 3.25.2 release changes the environment drop guard to warn and continue by default, with CONVOX_ENV_DROP_GUARD=strict to restore the blocking behavior. The 3.25.1 release extends per-app instance cost tracking to Azure racks with refreshed price tables for both AWS and Azure, adds Karpenter NodeOverlay support for fractional-GPU instance families and custom price modeling, expands convox run with six new pod customization flags, adds a CLI check that flags builds which would drop environment variables set in the running release, and propagates renewed certificates to app namespaces automatically.

3.25.0

Released: 2026-07-08

Updates

Upgraded Kubernetes to v1.35 across all providers (EKS, GKE, AKS, DOKS)
Updated kube-proxy to v1.35.3
Updated VPC CNI to v1.22.3
Updated CoreDNS to v1.14.3
Updated EBS CSI Driver to v1.62.0
Updated EFS CSI Driver to v3.3.0
Updated Cluster Autoscaler to v1.35.0

View on GitHub

3.25.1

Released: 2026-07-17

Feature Additions

Added per-app instance cost tracking for Azure racks via the new cost_tracking_enable rack parameter, enabling convox cost per-service breakdowns, Console cost dashboards, and per-app monthly budget caps with convox budget set, with automatic spot node pool detection and discounting
Added the karpenter_node_overlays_config rack parameter for creating Karpenter NodeOverlays, which can advertise extended resources such as nvidia.com/gpu on fractional-GPU instance families (g6f, gr6f) or adjust the prices Karpenter uses in its provisioning cost model
Added six optional convox run flags for per-invocation pod customization: --termination-grace, --annotations, --labels, --use-service-lifecycle, --node-affinity, and --tolerations
Added a CLI preflight check on convox build, convox deploy, and convox test that stops a build from dropping environment variables set in the running release, with a --force flag to bypass when the drop is intentional

Updates

Updated the Karpenter chart from 1.10.0 to 1.13.1, adding Kubernetes 1.35 and 1.36 support. Racks with Karpenter enabled may see a one-time gradual node replacement after this update, depending on whether the chart change alters the NodePool drift hash. Any replacement is graceful and bounded by karpenter_disruption_budget_nodes (default 10%); lower that value before updating for a slower roll
Expanded the AWS and Azure instance price tables to 489 instance types each, covering arm64 Graviton, AMD, network-optimized, storage-optimized, high-memory, and GPU families, with AWS prices refreshed to current us-east-1 Linux on-demand list prices
Run and timer pods now honor the service's termination.grace setting from convox.yml, matching the behavior of service deployments

Fixes

Renewed Let's Encrypt certificates now propagate automatically from the rack namespace to app namespaces, so services referencing a certificate through the certificate: option always serve the current certificate without requiring a new deploy

View on GitHub

3.25.2

Released: 2026-07-21

Updates

The environment drop guard added in 3.25.1 now warns and continues instead of blocking. When the newest Release drops environment variables that are still set in the running Release, convox build, convox deploy, and convox test print a WARNING: block to stderr naming the dropped variables and the Build proceeds. Set CONVOX_ENV_DROP_GUARD=strict to restore the blocking behavior, which fails with the same error text and the same exit code as 3.25.1. --force reduces the message to a one-line notice

Notes

3.25.2 is a CLI-only release with no Rack-side or Terraform change, so the new behavior arrives by updating the convox CLI rather than by running convox rack update

View on GitHub

3.25.3

Released: 2026-07-26

Feature Additions

Added the additional_node_groups_config rack parameter to GCP Racks, using the same JSON format as Azure. Each pool accepts type, disk, disk_type, capacity_type, min_size, max_size, label, dedicated, tags, and zones. GCP uses regional GKE clusters, so min_size and max_size apply per zone rather than as cluster totals
Added GPU node pools on GCP through the gpu_type and gpu_count fields of additional_node_groups_config, with the GKE-managed NVIDIA driver installed automatically
Added single-host Cloud TPU node pools on GCP through the tpu_topology field of additional_node_groups_config, which requires a TPU machine type and creates the pool with a COMPACT placement policy. Services request TPUs with scale.gpu.vendor: google, which maps to the google.com/tpu resource
Added the gpu_observability_enable, gpu_observability_chart_version, and dcgm_scrape_interval rack parameters to GCP Racks. Enabling GPU observability installs the NVIDIA DCGM exporter in kube-system on port 9400 and creates six Grafana dashboard ConfigMaps labeled grafana_dashboard=1 for sidecar discovery. GKE manages the device plugin, so nvidia_device_plugin_enable is not required on GCP. A GCP Rack does not run its own Prometheus; the exporter advertises itself with prometheus.io/* pod annotations for a Prometheus you install. Google Managed Prometheus does not read those annotations; it collects through a PodMonitoring resource you create yourself, and Convox does not create one
GCP nodes in the g2, a2, a3, a4, a4x, and g4 machine families now receive the convox.io/gpu-vendor label. N1 machines can attach GPUs individually and are not detectable by machine type, so they are not labeled
Added the cloudwatch_disable rack parameter for AWS. When enabled, the Rack stops creating, writing, and reading its own CloudWatch log groups, one per App (/convox/<rack>/<app>) plus the Rack group (/convox/<rack>/system), so the whole-App view convox logs and convox rack logs return empty. convox logs --service <name> reads Pod logs directly and is unchanged. Application logs continue to reach CloudWatch through Fluentd unless fluentd_disable is also set; the two together are the configuration for routing logs entirely elsewhere
Added the karpenter_build_disruption_budget_nodes rack parameter for AWS, default 100%, which sets the disruption budget for the Empty reason on the Karpenter build NodePool so idle build nodes are reclaimed promptly. Drift and underutilization disruption on that pool remains capped at 10%
Added multi-arch App image builds. On an AWS Rack with Karpenter enabled whose workload architectures span both amd64 and arm64 (across karpenter_arch and the arch field of additional_karpenter_nodepools_config), a Build now produces a multi-arch image index instead of a single-architecture image. Single-architecture Racks are unchanged
The BuildArch App parameter is now accepted by the Rack. Setting it pins an App's builds to amd64 or arm64, both for the architecture of the produced image and, on Racks with dedicated build nodes enabled, for build pod placement

Updates

AWS additional node groups and additional build node groups are now created before the old group is destroyed, so the replacement group is ready before the old one is removed. That ordering is not a guarantee that total capacity holds steady throughout a replacing edit, so leave headroom for pods to reschedule. Editing only the tags of an additional node group no longer forces a replacement; the tags are updated in place and the launch template change rolls the nodes gracefully
Raising min_size on an AWS additional node group now raises the live EKS desired size, and the maximum size when needed, before terraform apply runs, and waits for that scaling update to finish, so the apply no longer fails when the requested minimum exceeds the group's current desired size. The reconcile runs on Rack install, parameter updates, and version updates, and prints a NOTICE: line when it acts
Convox-owned Helm releases left in a pending-install, pending-upgrade, or pending-rollback state by an interrupted apply are now cleared before the next terraform apply on AWS Racks, so a Rack does not stay stuck behind a stranded release lock. Only Convox releases are considered (aws-lbc, karpenter, karpenter-crd, keda, vpa, dcgm-exporter, nvidia-device-plugin, contour, contour-internal), and only when they have been pending for more than 15 minutes
The zones and disk_type fields of additional_node_groups_config are now preserved by the CLI. Previously the CLI re-encoded the parameter through a structure that did not carry them, so those keys were dropped before reaching Terraform even on providers whose modules read them
karpenter_disruption_budget_nodes, karpenter_build_disruption_budget_nodes, and the disruption_budget_nodes field of additional_karpenter_nodepools_config now reject percentages above 100, so an out-of-range value fails in the CLI instead of at terraform apply
capacity_type inside additional_node_groups_config is now validated per provider. AWS accepts ON_DEMAND and SPOT; Azure and the other providers additionally accept Regular and Spot. Previously all four values were accepted on every provider, and the Azure spellings reached AWS, where they are invalid
Duplicate id values in additional_node_groups_config and additional_build_groups_config are now rejected instead of colliding on the Terraform for_each key
convox rack params set now rejects Karpenter instance family lists that cannot satisfy the Rack's architecture, both for karpenter_instance_families against karpenter_arch and for karpenter_build_instance_families against build_node_type. The check runs only when Karpenter is enabled and one of the relevant parameters is being set
convox rack params set now warns when cloudwatch_disable and fluentd_disable are set inconsistently on AWS: enabling cloudwatch_disable alone still leaves Fluentd shipping application logs to CloudWatch, and enabling fluentd_disable alone still leaves an empty log group per App
Upgraded the build toolchain to Go 1.25.12 and refreshed golang.org/x/text to 0.39.0, golang.org/x/crypto to 0.53.0, and go-jose/go-jose to 4.1.4
Build source packaging and .dockerignore matching now use moby/go-archive and moby/patternmatcher, which removes the docker/docker, moby/buildkit, containerd, and runc dependencies from the convox binary. Build context exclusion now matches docker build for nested patterns: a .dockerignore that combines a wildcard directory exclude with a re-include underneath it previously leaked files under the excluded directory into the build context. Patterns containing regex characters such as +, (, ), and | are now treated literally. This ships in the CLI, so it applies once you update the convox CLI, independent of your Rack version

Fixes

Fixed convox rack install metal failing at terraform validate. The metal router module used the key_algorithm argument removed in version 4 of the TLS provider and did not declare the provider
convox builds export and convox builds import now copy every platform in an image manifest instead of only the default one, so exporting and re-importing a multi-arch Build preserves all architectures
A failed image push during convox builds import now fails the import instead of being reported as success

Notes

Terraform module changes and Rack control-plane images arrive with convox rack update 3.25.3, but the changes that live in the convox binary do not. The stranded Helm release cleanup and the node group desired-size reconcile, and the new parameter names and validators, arrive only with an updated convox CLI, which for a Console-managed Rack means a Console deploy carrying that CLI version. A pre-3.25.3 CLI rejects the new parameter names outright
A single non-default karpenter_arch still builds native to the build node, so pair karpenter_arch=arm64 with a Graviton build_node_type
Containerized resource images are not architecture-aware
The BuildArch App parameter was announced in 3.24.1 but was not registered by the Rack until 3.25.3, so on earlier Racks the value was accepted and discarded
GCP additional node pools are new in 3.25.3. Downgrading a GCP Rack below 3.25.3 destroys every additional node pool, because the older cluster module does not declare the resource. Parameter reconciliation also drops the stored value, so a Console-managed Rack has it re-supplied on the way back up while a self-managed Rack has to set it again. See additional_node_groups_config
Set gpu_observability_enable=false and let that apply finish before downgrading a GCP Rack below 3.25.3. Downgrading with the exporter still enabled fails the apply with Error: Provider configuration not present, because the helm provider is new to the GCP cluster module in this release. See gpu_observability_enable

Thanks to @beastawakens for the three user-submitted GCP pull requests in this release.

View on GitHub

3.25 Releases

3.25.0

Released: 2026-07-08

Updates

Upgraded Kubernetes to v1.35 across all providers (EKS, GKE, AKS, DOKS)
Updated kube-proxy to v1.35.3
Updated VPC CNI to v1.22.3
Updated CoreDNS to v1.14.3
Updated EBS CSI Driver to v1.62.0
Updated EFS CSI Driver to v3.3.0
Updated Cluster Autoscaler to v1.35.0

View on GitHub

3.25.1

Released: 2026-07-17

Feature Additions

Added per-app instance cost tracking for Azure racks via the new cost_tracking_enable rack parameter, enabling convox cost per-service breakdowns, Console cost dashboards, and per-app monthly budget caps with convox budget set, with automatic spot node pool detection and discounting
Added the karpenter_node_overlays_config rack parameter for creating Karpenter NodeOverlays, which can advertise extended resources such as nvidia.com/gpu on fractional-GPU instance families (g6f, gr6f) or adjust the prices Karpenter uses in its provisioning cost model
Added six optional convox run flags for per-invocation pod customization: --termination-grace, --annotations, --labels, --use-service-lifecycle, --node-affinity, and --tolerations
Added a CLI preflight check on convox build, convox deploy, and convox test that stops a build from dropping environment variables set in the running release, with a --force flag to bypass when the drop is intentional

Updates

Updated the Karpenter chart from 1.10.0 to 1.13.1, adding Kubernetes 1.35 and 1.36 support. Racks with Karpenter enabled may see a one-time gradual node replacement after this update, depending on whether the chart change alters the NodePool drift hash. Any replacement is graceful and bounded by karpenter_disruption_budget_nodes (default 10%); lower that value before updating for a slower roll
Expanded the AWS and Azure instance price tables to 489 instance types each, covering arm64 Graviton, AMD, network-optimized, storage-optimized, high-memory, and GPU families, with AWS prices refreshed to current us-east-1 Linux on-demand list prices
Run and timer pods now honor the service's termination.grace setting from convox.yml, matching the behavior of service deployments

Fixes

Renewed Let's Encrypt certificates now propagate automatically from the rack namespace to app namespaces, so services referencing a certificate through the certificate: option always serve the current certificate without requiring a new deploy

View on GitHub

3.25.2

Released: 2026-07-21

Updates

The environment drop guard added in 3.25.1 now warns and continues instead of blocking. When the newest Release drops environment variables that are still set in the running Release, convox build, convox deploy, and convox test print a WARNING: block to stderr naming the dropped variables and the Build proceeds. Set CONVOX_ENV_DROP_GUARD=strict to restore the blocking behavior, which fails with the same error text and the same exit code as 3.25.1. --force reduces the message to a one-line notice

Notes

3.25.2 is a CLI-only release with no Rack-side or Terraform change, so the new behavior arrives by updating the convox CLI rather than by running convox rack update

View on GitHub

3.25.3

Released: 2026-07-26

Feature Additions

Added the additional_node_groups_config rack parameter to GCP Racks, using the same JSON format as Azure. Each pool accepts type, disk, disk_type, capacity_type, min_size, max_size, label, dedicated, tags, and zones. GCP uses regional GKE clusters, so min_size and max_size apply per zone rather than as cluster totals
Added GPU node pools on GCP through the gpu_type and gpu_count fields of additional_node_groups_config, with the GKE-managed NVIDIA driver installed automatically
Added single-host Cloud TPU node pools on GCP through the tpu_topology field of additional_node_groups_config, which requires a TPU machine type and creates the pool with a COMPACT placement policy. Services request TPUs with scale.gpu.vendor: google, which maps to the google.com/tpu resource
Added the gpu_observability_enable, gpu_observability_chart_version, and dcgm_scrape_interval rack parameters to GCP Racks. Enabling GPU observability installs the NVIDIA DCGM exporter in kube-system on port 9400 and creates six Grafana dashboard ConfigMaps labeled grafana_dashboard=1 for sidecar discovery. GKE manages the device plugin, so nvidia_device_plugin_enable is not required on GCP. A GCP Rack does not run its own Prometheus; the exporter advertises itself with prometheus.io/* pod annotations for a Prometheus you install. Google Managed Prometheus does not read those annotations; it collects through a PodMonitoring resource you create yourself, and Convox does not create one
GCP nodes in the g2, a2, a3, a4, a4x, and g4 machine families now receive the convox.io/gpu-vendor label. N1 machines can attach GPUs individually and are not detectable by machine type, so they are not labeled
Added the cloudwatch_disable rack parameter for AWS. When enabled, the Rack stops creating, writing, and reading its own CloudWatch log groups, one per App (/convox/<rack>/<app>) plus the Rack group (/convox/<rack>/system), so the whole-App view convox logs and convox rack logs return empty. convox logs --service <name> reads Pod logs directly and is unchanged. Application logs continue to reach CloudWatch through Fluentd unless fluentd_disable is also set; the two together are the configuration for routing logs entirely elsewhere
Added the karpenter_build_disruption_budget_nodes rack parameter for AWS, default 100%, which sets the disruption budget for the Empty reason on the Karpenter build NodePool so idle build nodes are reclaimed promptly. Drift and underutilization disruption on that pool remains capped at 10%
Added multi-arch App image builds. On an AWS Rack with Karpenter enabled whose workload architectures span both amd64 and arm64 (across karpenter_arch and the arch field of additional_karpenter_nodepools_config), a Build now produces a multi-arch image index instead of a single-architecture image. Single-architecture Racks are unchanged
The BuildArch App parameter is now accepted by the Rack. Setting it pins an App's builds to amd64 or arm64, both for the architecture of the produced image and, on Racks with dedicated build nodes enabled, for build pod placement

Updates

AWS additional node groups and additional build node groups are now created before the old group is destroyed, so the replacement group is ready before the old one is removed. That ordering is not a guarantee that total capacity holds steady throughout a replacing edit, so leave headroom for pods to reschedule. Editing only the tags of an additional node group no longer forces a replacement; the tags are updated in place and the launch template change rolls the nodes gracefully
Raising min_size on an AWS additional node group now raises the live EKS desired size, and the maximum size when needed, before terraform apply runs, and waits for that scaling update to finish, so the apply no longer fails when the requested minimum exceeds the group's current desired size. The reconcile runs on Rack install, parameter updates, and version updates, and prints a NOTICE: line when it acts
Convox-owned Helm releases left in a pending-install, pending-upgrade, or pending-rollback state by an interrupted apply are now cleared before the next terraform apply on AWS Racks, so a Rack does not stay stuck behind a stranded release lock. Only Convox releases are considered (aws-lbc, karpenter, karpenter-crd, keda, vpa, dcgm-exporter, nvidia-device-plugin, contour, contour-internal), and only when they have been pending for more than 15 minutes
The zones and disk_type fields of additional_node_groups_config are now preserved by the CLI. Previously the CLI re-encoded the parameter through a structure that did not carry them, so those keys were dropped before reaching Terraform even on providers whose modules read them
karpenter_disruption_budget_nodes, karpenter_build_disruption_budget_nodes, and the disruption_budget_nodes field of additional_karpenter_nodepools_config now reject percentages above 100, so an out-of-range value fails in the CLI instead of at terraform apply
capacity_type inside additional_node_groups_config is now validated per provider. AWS accepts ON_DEMAND and SPOT; Azure and the other providers additionally accept Regular and Spot. Previously all four values were accepted on every provider, and the Azure spellings reached AWS, where they are invalid
Duplicate id values in additional_node_groups_config and additional_build_groups_config are now rejected instead of colliding on the Terraform for_each key
convox rack params set now rejects Karpenter instance family lists that cannot satisfy the Rack's architecture, both for karpenter_instance_families against karpenter_arch and for karpenter_build_instance_families against build_node_type. The check runs only when Karpenter is enabled and one of the relevant parameters is being set
convox rack params set now warns when cloudwatch_disable and fluentd_disable are set inconsistently on AWS: enabling cloudwatch_disable alone still leaves Fluentd shipping application logs to CloudWatch, and enabling fluentd_disable alone still leaves an empty log group per App
Upgraded the build toolchain to Go 1.25.12 and refreshed golang.org/x/text to 0.39.0, golang.org/x/crypto to 0.53.0, and go-jose/go-jose to 4.1.4
Build source packaging and .dockerignore matching now use moby/go-archive and moby/patternmatcher, which removes the docker/docker, moby/buildkit, containerd, and runc dependencies from the convox binary. Build context exclusion now matches docker build for nested patterns: a .dockerignore that combines a wildcard directory exclude with a re-include underneath it previously leaked files under the excluded directory into the build context. Patterns containing regex characters such as +, (, ), and | are now treated literally. This ships in the CLI, so it applies once you update the convox CLI, independent of your Rack version

Fixes

Fixed convox rack install metal failing at terraform validate. The metal router module used the key_algorithm argument removed in version 4 of the TLS provider and did not declare the provider
convox builds export and convox builds import now copy every platform in an image manifest instead of only the default one, so exporting and re-importing a multi-arch Build preserves all architectures
A failed image push during convox builds import now fails the import instead of being reported as success

Notes

Terraform module changes and Rack control-plane images arrive with convox rack update 3.25.3, but the changes that live in the convox binary do not. The stranded Helm release cleanup and the node group desired-size reconcile, and the new parameter names and validators, arrive only with an updated convox CLI, which for a Console-managed Rack means a Console deploy carrying that CLI version. A pre-3.25.3 CLI rejects the new parameter names outright
A single non-default karpenter_arch still builds native to the build node, so pair karpenter_arch=arm64 with a Graviton build_node_type
Containerized resource images are not architecture-aware
The BuildArch App parameter was announced in 3.24.1 but was not registered by the Rack until 3.25.3, so on earlier Racks the value was accepted and discarded
GCP additional node pools are new in 3.25.3. Downgrading a GCP Rack below 3.25.3 destroys every additional node pool, because the older cluster module does not declare the resource. Parameter reconciliation also drops the stored value, so a Console-managed Rack has it re-supplied on the way back up while a self-managed Rack has to set it again. See additional_node_groups_config
Set gpu_observability_enable=false and let that apply finish before downgrading a GCP Rack below 3.25.3. Downgrading with the exporter still enabled fails the apply with Error: Provider configuration not present, because the helm provider is new to the GCP cluster module in this release. See gpu_observability_enable

Thanks to @beastawakens for the three user-submitted GCP pull requests in this release.

View on GitHub

3.25 Releases

3.25.0

3.25.1

3.25.2

3.25.3

See Also

3.25 Releases

3.25.0

3.25.1

3.25.2

3.25.3

See Also