A Terraform provider for Dokploy
Plugin-framework provider covering projects, environments, applications, Compose stacks, managed databases, domains, mounts, ports, redirects, basic auth, registries, SSH keys, certificates and backup destinations, over Dokploy's tRPC-over-REST API. The shim package exposes the provider to other Go modules, which is how pulumi-dokploy bridges it.
This commit is contained in:
671
.agents/skills/terraform-stacks/references/troubleshooting.md
Normal file
671
.agents/skills/terraform-stacks/references/troubleshooting.md
Normal file
@@ -0,0 +1,671 @@
|
||||
# Troubleshooting Reference
|
||||
|
||||
Common issues and solutions when working with Terraform Stacks.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Configuration Issues](#configuration-issues)
|
||||
2. [Deployment Issues](#deployment-issues)
|
||||
3. [Provider and Authentication Issues](#provider-and-authentication-issues)
|
||||
4. [Module Compatibility Issues](#module-compatibility-issues)
|
||||
5. [State and Dependency Issues](#state-and-dependency-issues)
|
||||
6. [API and CLI Issues](#api-and-cli-issues)
|
||||
|
||||
## Configuration Issues
|
||||
|
||||
### Circular Dependencies
|
||||
|
||||
**Issue:** Component A references Component B, and Component B references Component A.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Cycle detected in component dependencies
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Break the circular reference** by refactoring components:
|
||||
|
||||
```hcl
|
||||
# Before (circular dependency)
|
||||
component "vpc" {
|
||||
source = "./modules/vpc"
|
||||
inputs = {
|
||||
security_group_id = component.app.security_group_id # References app
|
||||
}
|
||||
}
|
||||
|
||||
component "app" {
|
||||
source = "./modules/app"
|
||||
inputs = {
|
||||
vpc_id = component.vpc.vpc_id # References vpc
|
||||
}
|
||||
}
|
||||
|
||||
# After (broken circular reference)
|
||||
component "vpc" {
|
||||
source = "./modules/vpc"
|
||||
inputs = {
|
||||
# Remove reference to app
|
||||
}
|
||||
}
|
||||
|
||||
component "security_group" {
|
||||
source = "./modules/security-group"
|
||||
inputs = {
|
||||
vpc_id = component.vpc.vpc_id
|
||||
}
|
||||
}
|
||||
|
||||
component "app" {
|
||||
source = "./modules/app"
|
||||
inputs = {
|
||||
vpc_id = component.vpc.vpc_id
|
||||
security_group_id = component.security_group.id
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Use intermediate components** to break the dependency chain
|
||||
3. **Refactor modules** to remove the circular dependency at the module level
|
||||
|
||||
### Validation Errors on Variables
|
||||
|
||||
**Issue:** Variable block validation errors during `terraform stacks validate`.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Unsupported argument
|
||||
on variables.tfcomponent.hcl line 5:
|
||||
5: validation {
|
||||
|
||||
Validation blocks are not supported in Stack configurations
|
||||
```
|
||||
|
||||
**Solution:** Remove `validation` blocks from variable declarations. Stacks do not support validation blocks:
|
||||
|
||||
```hcl
|
||||
# Incorrect
|
||||
variable "instance_count" {
|
||||
type = number
|
||||
validation {
|
||||
condition = var.instance_count > 0
|
||||
error_message = "Instance count must be positive"
|
||||
}
|
||||
}
|
||||
|
||||
# Correct
|
||||
variable "instance_count" {
|
||||
type = number
|
||||
description = "Number of instances (must be positive)"
|
||||
}
|
||||
```
|
||||
|
||||
Move validation logic into the underlying modules if needed.
|
||||
|
||||
### Missing Type in Variable Declarations
|
||||
|
||||
**Issue:** Variables fail validation when `type` is not specified.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Missing required argument
|
||||
on variables.tfcomponent.hcl line 3:
|
||||
3: variable "region" {
|
||||
|
||||
The argument "type" is required in Stack variable declarations
|
||||
```
|
||||
|
||||
**Solution:** Always specify `type` for variables - it's required in Stacks (unlike traditional Terraform):
|
||||
|
||||
```hcl
|
||||
# Incorrect
|
||||
variable "region" {
|
||||
default = "us-west-1"
|
||||
}
|
||||
|
||||
# Correct
|
||||
variable "region" {
|
||||
type = string
|
||||
default = "us-west-1"
|
||||
}
|
||||
```
|
||||
|
||||
### Provider Configuration in Modules
|
||||
|
||||
**Issue:** Modules with embedded provider blocks cause errors.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Provider configuration not allowed in module
|
||||
|
||||
Modules used with Terraform Stacks cannot contain provider blocks
|
||||
```
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. **Remove provider blocks from modules** - configure providers in Stack configuration instead
|
||||
2. **Use modules that don't contain provider blocks** (most public registry modules are compatible)
|
||||
3. **Fork and modify modules** if necessary to remove provider blocks
|
||||
|
||||
## Deployment Issues
|
||||
|
||||
### Cannot Destroy Deployment from UI
|
||||
|
||||
**Issue:** The HCP Terraform UI doesn't provide an option to destroy Stack deployments.
|
||||
|
||||
**Why:** Stack deployment destruction is only available through configuration, not the UI.
|
||||
|
||||
**Solution:** Set `destroy = true` in the deployment block and upload the configuration:
|
||||
|
||||
```hcl
|
||||
deployment "old_environment" {
|
||||
inputs = {
|
||||
aws_region = "us-west-1"
|
||||
instance_count = 2
|
||||
role_arn = local.role_arn
|
||||
identity_token = identity_token.aws.jwt
|
||||
}
|
||||
|
||||
destroy = true # Marks deployment for destruction
|
||||
}
|
||||
```
|
||||
|
||||
**Workflow:**
|
||||
|
||||
1. Add `destroy = true` to the deployment block
|
||||
2. Run `terraform stacks configuration upload`
|
||||
3. HCP Terraform creates a destroy run automatically
|
||||
4. Approve the destroy run (if auto-approve is not configured)
|
||||
5. After destruction completes, remove the deployment block entirely
|
||||
6. Upload configuration again to clean up the deployment definition
|
||||
|
||||
**Important:** You cannot destroy deployments from the UI. This is by design to prevent accidental destruction.
|
||||
|
||||
### Deployment Stuck in "Planning" State
|
||||
|
||||
**Issue:** Deployment remains in "planning" state indefinitely.
|
||||
|
||||
**Possible Causes:**
|
||||
|
||||
1. **Provider authentication failed** - Check OIDC configuration and IAM roles
|
||||
2. **Module download failed** - Verify module sources are accessible
|
||||
3. **Provider version conflict** - Check `.terraform.lock.hcl` matches required providers
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
```bash
|
||||
# Get deployment step diagnostics
|
||||
terraform stacks deployment-run list
|
||||
# Note the run ID, then:
|
||||
curl -s -H "Authorization: Bearer $TOKEN" \
|
||||
"https://app.terraform.io/api/v2/stack-deployment-runs/{run-id}/stack-deployment-steps" | \
|
||||
jq '.data[] | {id, status: .attributes.status, component: .attributes["component-name"]}'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Check diagnostics for the stuck step
|
||||
2. Verify provider authentication is configured correctly
|
||||
3. Ensure all module sources are accessible
|
||||
4. Check provider lock file matches required providers
|
||||
|
||||
### Deployment Requires Approval But No Approval Prompt
|
||||
|
||||
**Issue:** Deployment is waiting for approval but CLI doesn't show approval prompt.
|
||||
|
||||
**Why:** CLI monitoring commands are non-blocking and don't automatically prompt for approval.
|
||||
|
||||
**Solution:**
|
||||
|
||||
**Option 1: Approve via CLI**
|
||||
```bash
|
||||
# Approve all pending plans in a deployment run
|
||||
terraform stacks deployment-run approve-all-plans -deployment-run-id=sdr-ABC123
|
||||
|
||||
# Or approve all plans in a deployment group
|
||||
terraform stacks deployment-group approve-all-plans -deployment-group=canary
|
||||
```
|
||||
|
||||
**Option 2: Configure auto-approve** (Premium feature)
|
||||
```hcl
|
||||
deployment_auto_approve "safe_changes" {
|
||||
deployment_group = deployment_group.canary
|
||||
|
||||
check {
|
||||
condition = context.plan.applyable
|
||||
reason = "Plan must be successful"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Provider and Authentication Issues
|
||||
|
||||
### OIDC Authentication Failing
|
||||
|
||||
**Issue:** Provider authentication fails with OIDC/workload identity.
|
||||
|
||||
**Error Messages:**
|
||||
```
|
||||
Error: Error assuming role with web identity
|
||||
Error: Failed to retrieve credentials
|
||||
Error: Invalid identity token
|
||||
```
|
||||
|
||||
**Diagnosis Steps:**
|
||||
|
||||
1. **Verify identity token configuration:**
|
||||
|
||||
```hcl
|
||||
# Check identity_token block exists
|
||||
identity_token "aws" {
|
||||
audience = ["aws.workload.identity"]
|
||||
}
|
||||
|
||||
# Check deployment references the token
|
||||
deployment "production" {
|
||||
inputs = {
|
||||
identity_token = identity_token.aws.jwt
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Verify provider configuration:**
|
||||
|
||||
```hcl
|
||||
provider "aws" "this" {
|
||||
config {
|
||||
region = var.aws_region
|
||||
assume_role_with_web_identity {
|
||||
role_arn = var.role_arn
|
||||
web_identity_token = var.identity_token
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **Check IAM role trust policy:**
|
||||
|
||||
**AWS - Verify trust policy includes HCP Terraform:**
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": {
|
||||
"Federated": "arn:aws:iam::<account-id>:oidc-provider/app.terraform.io"
|
||||
},
|
||||
"Action": "sts:AssumeRoleWithWebIdentity",
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"app.terraform.io:aud": "aws.workload.identity"
|
||||
},
|
||||
"StringLike": {
|
||||
"app.terraform.io:sub": "organization:<org-name>:project:<project-name>:stack:<stack-name>:deployment:<deployment-name>"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Azure - Verify federated credential:**
|
||||
- Application ID matches the one in provider configuration
|
||||
- Subject matches: `organization:<org>:project:<project>:stack:<stack>:deployment:<deployment>`
|
||||
- Issuer is `https://app.terraform.io`
|
||||
|
||||
**GCP - Verify workload identity pool:**
|
||||
- Provider configuration includes correct workload identity provider
|
||||
- Service account has necessary IAM permissions
|
||||
- Attribute mapping includes `google.subject` from token claims
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Fix IAM role trust policy to include correct HCP Terraform OIDC provider
|
||||
2. Ensure audience matches between identity_token block and IAM trust policy
|
||||
3. Verify subject pattern matches your organization/project/stack/deployment names
|
||||
4. Check that the role_arn is correct in provider configuration
|
||||
|
||||
### Provider Version Lock File Issues
|
||||
|
||||
**Issue:** Provider version conflicts or "could not retrieve provider" errors.
|
||||
|
||||
**Error Messages:**
|
||||
```
|
||||
Error: Failed to install provider
|
||||
Error: Provider version not found
|
||||
Error: Checksum mismatch for provider
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Regenerate provider lock file:**
|
||||
|
||||
```bash
|
||||
terraform stacks providers-lock
|
||||
```
|
||||
|
||||
2. **Add additional platforms** (if deploying from different OS):
|
||||
|
||||
```bash
|
||||
terraform stacks providers-lock \
|
||||
-platform=linux_amd64 \
|
||||
-platform=darwin_amd64 \
|
||||
-platform=darwin_arm64
|
||||
```
|
||||
|
||||
3. **Verify required_providers block:**
|
||||
|
||||
```hcl
|
||||
required_providers {
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = "~> 5.7.0" # Ensure version constraint is valid
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. **Commit `.terraform.lock.hcl`** to version control
|
||||
|
||||
## Module Compatibility Issues
|
||||
|
||||
### Public Registry Module Errors
|
||||
|
||||
**Issue:** Modules from the Terraform public registry cause errors during plan or apply.
|
||||
|
||||
**Common Errors:**
|
||||
```
|
||||
Error: Unsupported attribute
|
||||
Error: Invalid reference
|
||||
Error: Missing required argument
|
||||
```
|
||||
|
||||
**Known Problematic Modules:**
|
||||
- `terraform-aws-modules/alb/aws` - Some versions have compatibility issues
|
||||
- `terraform-aws-modules/ecs-service/aws` - May have issues with certain configurations
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Test modules in dev deployment first** before using in production
|
||||
|
||||
2. **Check module compatibility** by reviewing recent issues on the module repository
|
||||
|
||||
3. **Use specific module versions** rather than latest:
|
||||
|
||||
```hcl
|
||||
component "alb" {
|
||||
source = "terraform-aws-modules/alb/aws"
|
||||
version = "8.7.0" # Use specific version known to work
|
||||
# ...
|
||||
}
|
||||
```
|
||||
|
||||
4. **Consider using raw resources** for critical infrastructure:
|
||||
|
||||
```hcl
|
||||
# Instead of using a module that has issues
|
||||
component "alb" {
|
||||
source = "./modules/alb" # Create local module with raw resources
|
||||
# ...
|
||||
}
|
||||
```
|
||||
|
||||
5. **Fork and fix modules** if you have the resources to maintain them
|
||||
|
||||
6. **Report compatibility issues** to module maintainers
|
||||
|
||||
### Local Module Not Found
|
||||
|
||||
**Issue:** Stack can't find local module sources.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Module not found
|
||||
Could not load module ./modules/vpc
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Verify module path is relative** to Stack root:
|
||||
|
||||
```hcl
|
||||
# Correct
|
||||
component "vpc" {
|
||||
source = "./modules/vpc"
|
||||
}
|
||||
|
||||
# Incorrect (absolute paths don't work)
|
||||
component "vpc" {
|
||||
source = "/Users/username/project/modules/vpc"
|
||||
}
|
||||
```
|
||||
|
||||
2. **Ensure module directory exists** with proper structure:
|
||||
|
||||
```
|
||||
my-stack/
|
||||
├── components.tfcomponent.hcl
|
||||
└── modules/
|
||||
└── vpc/
|
||||
├── main.tf
|
||||
├── variables.tf
|
||||
└── outputs.tf
|
||||
```
|
||||
|
||||
3. **Check file permissions** on module directories
|
||||
|
||||
## State and Dependency Issues
|
||||
|
||||
### Component Output Not Available
|
||||
|
||||
**Issue:** Component output is not available to referencing component.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Reference to unknown component
|
||||
Component "vpc" has not been defined
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Verify component exists** in configuration:
|
||||
|
||||
```hcl
|
||||
component "vpc" {
|
||||
source = "./modules/vpc"
|
||||
# Must define component before referencing it
|
||||
}
|
||||
|
||||
component "app" {
|
||||
source = "./modules/app"
|
||||
inputs = {
|
||||
vpc_id = component.vpc.vpc_id # Now valid
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Check output is defined in module:**
|
||||
|
||||
```hcl
|
||||
# In modules/vpc/outputs.tf
|
||||
output "vpc_id" {
|
||||
value = aws_vpc.main.id
|
||||
}
|
||||
```
|
||||
|
||||
3. **For components with for_each**, reference specific instance:
|
||||
|
||||
```hcl
|
||||
component "regional" {
|
||||
for_each = var.regions
|
||||
# ...
|
||||
}
|
||||
|
||||
component "app" {
|
||||
inputs = {
|
||||
# Correct - reference specific instance
|
||||
vpc_id = component.regional["us-west-1"].vpc_id
|
||||
|
||||
# Incorrect - can't reference for_each component directly
|
||||
# vpc_id = component.regional.vpc_id
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Deferred Changes Not Converging
|
||||
|
||||
**Issue:** Deployment with deferred changes doesn't complete after multiple iterations.
|
||||
|
||||
**Error Message:**
|
||||
```
|
||||
Error: Maximum deferred change iterations reached
|
||||
```
|
||||
|
||||
**Cause:** Dependency cycle or values that never stabilize.
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Review component dependencies** for logical cycles
|
||||
2. **Check for computed values that change on every run**
|
||||
3. **Refactor to break dependency chain**
|
||||
4. **Consider multi-stage deployments** if resources truly can't be created together
|
||||
|
||||
## API and CLI Issues
|
||||
|
||||
### Empty Diagnostics Response
|
||||
|
||||
**Issue:** API request for diagnostics returns empty results.
|
||||
|
||||
**Request:**
|
||||
```bash
|
||||
curl "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics"
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"data": []
|
||||
}
|
||||
```
|
||||
|
||||
**Solution:** Add required `stack_deployment_step_id` query parameter:
|
||||
|
||||
```bash
|
||||
curl "https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/stack-diagnostics?stack_deployment_step_id={step-id}"
|
||||
```
|
||||
|
||||
### Cannot Retrieve Stack Outputs
|
||||
|
||||
**Issue:** No CLI command to retrieve Stack outputs after deployment.
|
||||
|
||||
**Why:** Currently no direct CLI command for outputs retrieval.
|
||||
|
||||
**Solution:** Use the artifacts API endpoint:
|
||||
|
||||
```bash
|
||||
# Get final apply step ID first
|
||||
APPLY_STEP=$(terraform stacks deployment-run list --json | \
|
||||
jq -r '.[0].deployment_steps[] | select(.operation_type == "apply") | .id' | tail -1)
|
||||
|
||||
# Get outputs
|
||||
curl -L -s -H "Authorization: Bearer $TOKEN" \
|
||||
"https://app.terraform.io/api/v2/stack-deployment-steps/$APPLY_STEP/artifacts?name=apply-description" | \
|
||||
jq -r '.outputs | to_entries | .[] | "\(.key): \(.value.change.after)"'
|
||||
```
|
||||
|
||||
### CLI Watch Commands Hang in CI/CD
|
||||
|
||||
**Issue:** Commands like `terraform stacks deployment-run watch` never return in CI/CD pipelines.
|
||||
|
||||
**Why:** Watch commands stream output indefinitely and are designed for interactive use.
|
||||
|
||||
**Solution:** Use API polling instead of watch commands. See `api-monitoring.md` for complete workflow.
|
||||
|
||||
### Artifacts Endpoint Returns 404
|
||||
|
||||
**Issue:** Request to artifacts endpoint returns 404 Not Found.
|
||||
|
||||
**Possible Causes:**
|
||||
|
||||
1. **Step hasn't completed yet** - wait for step status to be "completed"
|
||||
2. **Wrong artifact name** - use one of: plan-description, plan-debug-log, apply-description, apply-debug-log
|
||||
3. **Invalid step ID** - verify step ID from deployment-steps endpoint
|
||||
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
# Check step status first
|
||||
curl -s -H "Authorization: Bearer $TOKEN" \
|
||||
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}" | \
|
||||
jq '.data.attributes.status'
|
||||
|
||||
# Only request artifacts when status is "completed"
|
||||
if [ "$STATUS" = "completed" ]; then
|
||||
curl -L -H "Authorization: Bearer $TOKEN" \
|
||||
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/artifacts?name=apply-description"
|
||||
fi
|
||||
```
|
||||
|
||||
### HTTP 307 Redirect Not Followed
|
||||
|
||||
**Issue:** Artifacts endpoint returns redirect response instead of artifact content.
|
||||
|
||||
**Why:** The endpoint returns HTTP 307 redirect to the actual artifact URL.
|
||||
|
||||
**Solution:** Configure HTTP client to follow redirects:
|
||||
|
||||
```bash
|
||||
# curl: Use -L flag
|
||||
curl -L -H "Authorization: Bearer $TOKEN" \
|
||||
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/artifacts?name=apply-description"
|
||||
|
||||
# Python requests: allow_redirects=True (default)
|
||||
import requests
|
||||
response = requests.get(url, headers=headers, allow_redirects=True)
|
||||
|
||||
# Node.js fetch: redirect: 'follow' (default)
|
||||
const response = await fetch(url, {
|
||||
headers: headers,
|
||||
redirect: 'follow'
|
||||
});
|
||||
```
|
||||
|
||||
## Getting Additional Help
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
For more detailed error information, enable debug logging:
|
||||
|
||||
```bash
|
||||
# CLI commands
|
||||
TF_LOG=DEBUG terraform stacks validate
|
||||
TF_LOG=DEBUG terraform stacks configuration upload
|
||||
|
||||
# API artifacts
|
||||
# Request the debug-log artifact instead of description
|
||||
curl -L -H "Authorization: Bearer $TOKEN" \
|
||||
"https://app.terraform.io/api/v2/stack-deployment-steps/{step-id}/artifacts?name=apply-debug-log"
|
||||
```
|
||||
|
||||
### Check HCP Terraform Status
|
||||
|
||||
If experiencing widespread issues, check HCP Terraform status page:
|
||||
- https://status.hashicorp.com
|
||||
|
||||
### Review Configuration Version
|
||||
|
||||
List recent configurations to identify when issues started:
|
||||
|
||||
```bash
|
||||
terraform stacks configuration list
|
||||
```
|
||||
|
||||
### Contact Support
|
||||
|
||||
For issues not covered here:
|
||||
1. Gather relevant error messages and diagnostics
|
||||
2. Note the configuration sequence number
|
||||
3. Include deployment run IDs
|
||||
4. Contact HashiCorp Support with details
|
||||
Reference in New Issue
Block a user