Contact Us 1-800-596-4880

Troubleshoot Migration Issues

Diagnose and resolve traffic switching, certificate, connectivity, app, VPC upgrade, and deployment issues encountered during CloudHub to CloudHub 2.0 migration.

Traffic Switching Issues

Resolve 502 errors, traffic-switching percentage failures, and DLB configuration problems during cutover.

502 Bad Gateway During Traffic Switch

Several distinct issues can cause a 502 error during or after traffic switching.

  • TLS protocol incompatibility: The DLB requires TLS 1.2 or higher as the upstream protocol when routing to CloudHub 2.0 apps. If configured with only TLS 1.1, traffic switching to CloudHub 2.0 fails with a 502 error. Incorrect ingress endpoint selection can also cause 404 errors after resolving the TLS issue.

  • Stale backend IP after a redeploy: CloudHub 2.0 Edge Ingress can briefly route to a replica’s previous (now-terminated) ClusterIP after a redeploy, causing connect timeouts and intermittent 502s until ingress routing refreshes.

  • Path-rewrite trailing slash: Ingress path rewrite can append an unexpected trailing slash on an exact-prefix match, breaking APIkit routing and returning a 502.

  • Listener bound to the wrong interface: If the app’s HTTP listener is bound to 127.0.0.1 instead of 0.0.0.0, the ingress controller can’t reach it, which surfaces as a 502.

  • Duplicate vanity domain paths: Two or more apps in the same private space sharing the same vanity domain and path cause ambiguous routing and intermittent 502s.

To troubleshoot the TLS cause.

  1. In Runtime Manager, select CloudHub > Load Balancers.

  2. Select your DLB and review the Upstream TLS settings.

  3. Verify that TLS 1.2 or higher is enabled (it can be added alongside TLS 1.1).

  4. In your CloudHub 2.0 application, go to the Ingress tab and select the correct DLB endpoint from the dropdown.

  5. Verify that the endpoint mapping matches your DLB vanity domain configuration.

For more information, see KA 005315578 and KA 005305510 (duplicate vanity domain paths).

Failed to Update Traffic Switching Percentage

Missing permissions, incorrect application configuration, or certificate migration issues can prevent the traffic switching percentage from being updated.

To troubleshoot this issue.

  1. Verify that the user has the CloudHub Network Administrator permission.

  2. Verify that both CloudHub 1.0 and CloudHub 2.0 applications are deployed and running.

  3. Verify that the CloudHub 2.0 application’s HTTP Listener is listening on port 8081.

  4. Enable Last-Mile Security only when your application has a TLS context.

  5. If migrating with a DLB, go to Runtime Manager > Private Spaces > your private space > Domains & TLS and verify that the DLB certificate was migrated. If it’s missing, upload it and create a TLS context.

The DLB requires a full chain (intermediate and root certificates). For the private space, split intermediate and root certificates into a single file and upload as CA Path Certificate.

For more information, see KA 005321445.

Certificate and TLS Issues

Diagnose missing DLB certificates and TLS handshake failures after upgrading to a private space.

Missing DLB Certificates Post-Upgrade

The VPC upgrade tool migrates DLB configurations on a best-effort basis. Due to architectural differences between the DLB and CloudHub 2.0 ingress controller, automated certificate migration isn’t guaranteed.

To troubleshoot this issue.

  1. In Runtime Manager, go to your private space > Domains & TLS tab.

  2. Manually upload your server certificates and private keys.

  3. Verify that all Subject Alternative Names (SANs) for vanity domains are correctly listed.

  4. Create TLS contexts for each vanity domain.

Split intermediate and root certificates into a separate file and upload as the CA Path Certificate.

For more information, see KA 005237268.

TLS Handshake Failure

The ingress load balancer doesn’t use the certificates, or the certificates use incompatible TLS 1.1 ciphers.

To troubleshoot this issue.

  1. Upload certificates to the ingress load balancer in the private space’s Domains & TLS tab.

  2. Verify that no TLS 1.1-only ciphers are in use.

    • ECDHE-ECDSA-AES128-SHA1

    • ECDHE-ECDSA-AES256-SHA1

    • ECDHE-RSA-AES128-SHA1

    • ECDHE-RSA-AES256-SHA1

  3. If these ciphers are present, reissue certificates with TLS 1.2+ compatible ciphers.

Connectivity Issues

Use this table to resolve connection, firewall, and DNS issues between CloudHub, CloudHub 2.0, and on-premises systems.

Symptom Likely Cause Resolution Reference

CloudHub 2.0 application can’t connect to CloudHub application

DLB allowlist missing infrastructure CIDR ranges.

Add these CIDR ranges to the DLB allowlist: 100.64.0.0/16, 100.66.0.0/16, 100.67.0.0/16, 100.68.0.0/16.

Upgrade a VPC to a Private Space

CloudHub 2.0 application can’t connect to another CloudHub 2.0 application

Incorrect private space firewall rules or missing internal network configuration.

Add an ingress rule allowing traffic from the local private network. Verify domain configuration.

Connection refused from on-premises

Firewall rules not migrated correctly to the private space.

Check firewall rules in the private space. Verify VPN and transit gateway routing tables.

DNS not resolving

DNS CNAME not updated to point to CloudHub 2.0 endpoints.

Update the DNS CNAME to point to the ingress load balancer endpoint.

Application Issues

Use this table to resolve deployment, property, scheduler, and performance issues in migrated applications.

Symptom Likely Cause Resolution Reference

Application doesn’t start

Incompatible runtime version, or the runtime is past end of support.

Deploy on a currently supported LTS or Edge runtime. If the deployment fails with Runtime Version not supported for this target, the runtime is past end of support or blocked (Mule 4.3.0 deploys are blocked on CloudHub 2.0).

Application properties missing

Secure properties not re-entered during migration.

Re-enter all secure property values in the CloudHub 2.0 deployment settings.

Scheduler not firing, or firing on every replica (duplicate executions)

Clustering isn’t enabled. Without clustering, every replica runs the schedule independently, which causes duplicate executions; a timezone or configuration difference can also prevent a schedule from firing.

Enable clustering so only the primary replica runs the schedule. Review scheduler component configuration for CloudHub 2.0 compatibility. See Managing App Schedules.

High latency after migration

Replica size under-provisioned.

Increase replica size or count. Compare with original CloudHub worker sizing.

Oracle Stored Procedure Dropping Connections After 8 Minutes

After migrating to CloudHub 2.0, long-running stored procedure calls can terminate without an error after about 8 minutes. An intermediate network component (load balancer, NAT gateway, or firewall) drops idle TCP connections without application-level error reporting.

To troubleshoot this issue.

  • Option 1: Set JVM argument oracle.net.keepAlive=true in Mule Runtime through Runtime Manager properties.

  • Option 2: Use a Generic connection in Database Connector with the keepAlive property in the JDBC connection URL:

    jdbc:oracle:thin:@//host:port/service?oracle.net.keepAlive=true

For more information, see KA 005321536.

VPC Upgrade Issues

Resolve eligibility, capacity, certificate, and deletion problems that can occur when upgrading a VPC to a private space.

Symptom Likely Cause Resolution

Upgrade tooling isn’t available on the Settings page

Application isn’t in an eligible VPC, or VPC hasn’t been upgraded yet.

Verify that the VPC meets eligibility requirements and has been upgraded to a private space first.

Upgrade fails with eligibility or capacity checks

VPC doesn’t meet eligibility requirements: ≥25% free space and 100+ available IPs per subnet; legacy VPNs configured; Direct Connect or VPC Peering in use; TLS 1.1 ciphers present; or default VPC enabled. TCP traffic is also not supported on upgraded spaces.

Verify all requirements per VPC Upgrade Eligibility. Migrate Direct Connect or VPC Peering to Transit Gateway or VPN first. Reissue TLS 1.1 certificates with TLS 1.2+ ciphers. If your app needs TCP, create a new private space manually instead.

DLB certificates missing after upgrade

TLS 1.1 ciphers present in the original certificate, or transfer failure during best-effort migration.

Re-upload certificates to the private space Domains & TLS tab. If they use TLS 1.1 ciphers, then reissue the certificates. See Missing DLB Certificates Post-Upgrade.

Upgrade tooling doesn’t support default VPC

Only named VPCs can be upgraded. The default VPC isn’t eligible.

Create a new private space manually. Deploy applications directly to the new private space.

Private space doesn’t appear after upgrade

Upgrade is still in progress, or the upgrade failed silently.

Allow up to 30 minutes for the upgrade to complete (see KA 005389730). If the private space still doesn’t appear, then check Runtime Manager > Private Spaces and contact MuleSoft Support.

Can’t delete VPC after upgrade

Migration not marked complete, or environments still associated with the VPC.

Mark the migration as complete for all applications in the VPC. Dissociate all environments. See KA 005115579.

For upgrade steps, see Upgrade a VPC to a Private Space.

Infrastructure Issues

Use this table to resolve private space, networking, DNS, and configuration problems during infrastructure migration.

Symptom Likely Cause Resolution Reference

CIDR block conflict during private space creation

CIDR block overlaps with existing on-premises or cloud networks.

Choose a non-overlapping CIDR block. Review existing allocations before creating.

Create a New Private Space Manually

VPN tunnel not establishing to new private space

Routing tables or security group configurations not updated for new private space.

Verify routing tables. Confirm VPN configuration matches private space network settings.

Establish Network Connectivity

Missing DLB certificates post-upgrade

TLS 1.1 cipher incompatibility or transfer failure during upgrade.

Re-upload certificates to private space. Validate supported ciphers.

Missing DLB Certificates Post-Upgrade

Firewall rules not migrated correctly

Ports other than 80, 443, and 30500–32500 are dropped during migration.

Manually recreate dropped firewall rules in the private space.

Configuring Firewall Rules to Manage Traffic to and from a Private Space

Cannot upgrade default VPC

Upgrade tooling doesn’t support default VPC.

Disable the default VPC configuration before proceeding with the upgrade.

Review Environment Association

Can’t delete VPC after migration

Migration not marked complete, or environment still associated.

Mark upgrade complete in Runtime Manager. Verify that no environments are associated.

KA 005115579

Application doesn’t start on CloudHub 2.0

Incompatible Mule runtime version. Mule 4.3.0 deploys are blocked on CloudHub 2.0, and runtimes past end of support are rejected with Runtime Version not supported for this target.

Deploy on a currently supported LTS or Edge runtime and rebuild the JAR. Verify minMuleVersion in mule-artifact.json.

Edge and LTS Releases for Mule

Connection refused from on-premises systems

Private space firewall rules don’t allow egress to on-premises destinations, or transit gateway/VPN routing not configured.

Add egress firewall rules for the on-premises CIDR range. Verify that the transit gateway attachment or VPN tunnel is active and that routing tables include the destination.

Configuring Firewall Rules to Manage Traffic to and from a Private Space

TLS handshake failure

Certificates not uploaded to the private space TLS context, or TLS 1.1 ciphers in use.

Upload certificates to the private space Domains & TLS tab. Verify that all certificates use TLS 1.2 or higher compatible ciphers.

Configuring Domains and Certificates (TLS Context) for a Private Space

Application properties missing after migration

Secure properties aren’t carried over automatically. Copied properties require at least one edit to persist.

Re-enter all secure property values in the CloudHub 2.0 application settings. Edit or add at least one property to persist all copied properties.

Application Properties

DNS not resolving for CloudHub 2.0 application

DNS CNAME not updated to point to the CloudHub 2.0 ingress endpoint.

Update the DNS CNAME to point to the private space’s ingress endpoint, not the application’s own URL: <space-id>.<region>.cloudhub.io for public access, or internal-<space-id>.<region>.cloudhub.io for internal access (where the region code varies by control plane, for example usa-e1 for US or deu-c1.n for EU Germany).

CloudHub 2.0 Networking Architecture

High latency after migration

Replica size under-provisioned compared to original CloudHub worker size.

Increase replica vCore size or replica count. Compare original worker sizing to current replica allocation using the vCore equivalency table.

CloudHub 2.0 Replicas

Scheduler not firing, or firing on every replica (duplicate executions)

Clustering determines scheduler behavior. With clustering enabled, only the primary replica runs the scheduler; clustering itself remains active-active, but schedulers and polling sources are primary-only. Without clustering, every replica runs the schedule independently, causing duplicate executions.

Enable clustering to avoid duplicate executions. Verify that scheduler configuration is compatible with CloudHub 2.0 clustering, and review timezone settings. See scheduler management docs for CloudHub 2.0-specific behavior.

Managing App Schedules

Connectivity Diagnostics

To diagnose connectivity issues between CloudHub platforms during migration, deploy the MuleSoft Network Tools application to your VPC or private space. Use these tests.

  • Socket test: Verify TCP connectivity between platforms (equivalent to a TCP handshake).

  • DNS test: Confirm DNS resolution for DLB and ingress endpoints.

  • cURL test: Validate HTTP or HTTPS endpoint accessibility end-to-end.

  • Certificate test: Inspect TLS certificates presented by an endpoint.

  • Cipher test: Identify which TLS ciphers a remote endpoint supports.

In CloudHub 2.0 private spaces, Ping isn’t supported due to security restrictions. Use TCP socket tests instead.

Additional Troubleshooting Scenarios

Use this table to address deployment states, timeouts, rolling updates, static IP changes, and VPN routing issues not covered elsewhere.

Symptom Likely Cause Resolution

Application deployment stuck in PENDING state, or replicas TERMINATED/RECOVERING

Insufficient cluster capacity or out-of-memory (OOM) errors at startup. The private space may not have enough available resources.

Check replica status and reason in Runtime Manager. Increase replica size or count, or reduce memory consumption in the application. Check if other applications in the private space are consuming resources.

"Unknown error" when deploying CH2 application after linking to CH1 app

The CH1 application was stopped before the link was fully established, causing the deployment to fail.

Keep the CH1 application STARTED until the link is complete and verified. See Step 4: Validate and Switch Traffic for proper sequencing.

504 (Gateway Timeout) errors during or after traffic switch

Client-side request takes longer than 300 seconds to send, or application takes longer than configured to respond. Read Response Timeout and max request timeout limits apply.

Verify client-side timeout is less than 300 seconds. Configure Read Response Timeout up to 3,600 seconds in the private space’s Advanced settings. For uploads or long-running processes, break requests into smaller chunks or use asynchronous patterns.

Brief 502 or 503 errors during PS rolling updates or patches

During scheduled monthly patches or replica rolling updates, the platform temporarily recycles infrastructure. This is expected behavior.

Schedule traffic-switching cutover to avoid monthly maintenance windows if possible. Expect brief connectivity gaps during rolling updates. Monitor and resume after infrastructure stabilizes.

Object Store v2 in cluster mode throws ConcurrentModificationException or persistent VM queues lost on restart

Object Store v2 or VM queues in clustered mode have known issues. VM queue storage is ephemeral and lost on redeploy or restart.

Migrate to Anypoint MQ for persistent queues and disable persistent VM queues. For Object Store, verify cluster serialization configuration. See Batch Processing for CH2 batch job behavior.

Outbound static IP addresses changed after migration; third-party systems reject traffic

Migrated private spaces have different outbound NAT IPs than CH1 apps. Third-party allowlists weren’t updated with the new CH2 private space IPs (KI-62898).

Identify all backend systems that allowlist by source IP. Update their allowlists to include the new private space outbound IPs before switching 100% of traffic. During migration, maintain both old and new IPs in allowlists.

VPN connection shows "Unavailable" after migration; traffic doesn’t flow

Duplicate or overlapping static routes exist on the VPN connection after migration. These can prevent proper route advertisement.

Check VPN routing tables for duplicate or overlapping routes. Remove duplicates. If issue persists, contact MuleSoft Support with your VPN connection ID and VPC/private space IDs.

Ping fails from private space to on-premises or external hosts

CloudHub 2.0 private spaces don’t support ICMP (ping). The firewall or network policy may block UDP-based tests as well.

Use TCP socket tests instead of ping. Deploy the Network Tools application to the private space and use its Socket test to verify connectivity.

DLB mTLS or URL-rewrite mapping rules are missing after migration

The VPC upgrade tool migrates these configurations on a best-effort basis. Some complex rules or custom configurations may not transfer automatically.

Manually recreate URL-rewrite rules as per-app path and path-rewrite settings in the Ingress tab. Reconfigure mTLS via the private space’s TLS contexts and truststores. Verify mappings match the original DLB configuration.