additional_node_groups_config

Description

The additional_node_groups_config parameter allows you to configure additional customized node groups for your cluster. This feature enables more granular control over your Kubernetes infrastructure by letting you define node groups with specific instance types, capacity types, scaling parameters, and custom labels.

When combined with the additional_build_groups_config parameter and node selector configurations, you can optimize workload placement, improve cost efficiency, and separate operational concerns within your cluster.

Default Value

The default value for additional_node_groups_config is an empty array.

Use Cases

• Workload-Specific Optimization: Create node groups tailored to specific workload requirements (e.g., CPU-intensive, memory-intensive, or batch processing workloads).
• Cost Optimization: Configure certain node groups to use spot instances for non-critical workloads while maintaining on-demand instances for mission-critical services.
• Isolation: Segregate workloads by dedicating specific node groups to particular services.
• Resource Efficiency: Run different workloads on appropriately sized instances for optimal resource utilization and cost efficiency.

Configuration Format

The additional_node_groups_config parameter takes a JSON array of node group configurations. Each node group configuration is a JSON object with the following fields:

* Important: Custom AMI configuration should be used with extreme caution. AMIs in EKS clusters have strict compatibility requirements, and improper configuration can lead to cluster update failures requiring manual intervention. Only use custom AMIs if you have specific compatibility requirements and thoroughly understand EKS node bootstrapping processes. We recommend testing in a non-production environment before implementation.

Setting Parameters

To set the additional_node_groups_config parameter, there are several methods:

Using a JSON File (Recommended)

$ convox rack params set additional_node_groups_config=/path/to/node-config.json -r rackName
Setting parameters... OK

The JSON file should be structured as follows:

[
  {
    "id": 101,
    "type": "t3.medium",
    "disk": 50,
    "capacity_type": "ON_DEMAND",
    "min_size": 1,
    "max_size": 3,
    "label": "app-workers",
    "tags": "environment=production,team=backend"
  },
  {
    "id": 102,
    "type": "m5.large",
    "disk": 100,
    "capacity_type": "SPOT",
    "min_size": 2,
    "max_size": 5,
    "label": "batch-workers",
    "ami_id": "ami-0123456789abcdef0",
    "tags": "environment=production,team=data,workload=batch"
  }
]

Important Note on AWS Rate Limits: When adding or removing multiple node groups, it’s recommended to modify no more than three node groups at a time to avoid hitting AWS API rate limits. If you receive a rate limit error during an update simply run the parameter set command again. The operation will resume from where it left off, creating the remaining node groups without duplicating the ones that were already successfully created.

Using a Raw JSON String

$ convox rack params set 'additional_node_groups_config=[{"id":101,"type":"t3.medium","disk":50,"capacity_type":"ON_DEMAND","min_size":1,"max_size":3,"label":"app-workers","tags":"environment=production,team=backend"}]' -r rackName
Setting parameters... OK

Node Group Identification and Tagging

Using the id Field

The id field ensures that node groups preserve their identity during configuration updates:

• Each node group should have a unique integer identifier
• Using the id field prevents unnecessary recreation of node groups when making changes to their configuration
• Without an id, Convox generates a random identifier that changes when configurations are updated, potentially causing disruptive node group replacements
• Consistent id values help maintain stable infrastructure during updates

Example configuration using the id field:

[
  {
    "id": 101,
    "type": "t3.medium",
    "label": "web-services",
    "min_size": 1,
    "max_size": 5
  }
]

Using the tags Field

The tags field allows you to add AWS tags to specific node groups:

• Tags help with cost allocation, resource organization, and compliance tracking
• Specify tags as comma-separated key-value pairs (e.g., "environment=production,team=backend")
• Tags are applied directly to the AWS node group resources
• Custom tags can be used with AWS cost management and reporting tools

Example configuration using the tags field:

[
  {
    "id": 101,
    "type": "t3.medium",
    "label": "web-services",
    "min_size": 1,
    "max_size": 5,
    "tags": "environment=production,team=frontend,tier=web"
  }
]

Using Node Groups with Services

To target specific services to run on particular node groups, use the nodeSelectorLabels field in your convox.yml file:

services:
  web:
    nodeSelectorLabels:
      convox.io/label: app-workers

This will ensure that the web service is scheduled only on nodes with the label convox.io/label: app-workers.

Additional Information

When using dedicated node groups (with dedicated: true), only services with matching node selector labels will be scheduled on those nodes. This provides strong isolation for workloads with specific requirements.

For build-specific node groups, see the additional_build_groups_config parameter.

Properly configured node groups can significantly improve cluster efficiency, resource utilization, and cost optimization while providing the right resource profiles for different workload types.