Resolved. CU side fiber optic cable not seated securely.

DMIT is currently upgrading the connection capacity with China Unicom. It takes a while to finish the upgrade. That's the reason why redundant interface didn't help on this fault.

LAX.EB:
Observed China Unicom AS10099-9929 latency and packet loss increases.
We've reached vendors about this matter.

DMIT both circuits with China Unicom are both in good conditions.

Based on the following observation, we thought that it could be cable fault in Unicom's network. It leads to China Unicom AS10099-9929 loss capacity, which leads to U.S. -> China direction congested.

• LAX.Pro has not been impacted;
• LAX. Pro shares inbound with LAX.EB ( China Unicom -> DMIT directions. )
• All China Unicom customers in Los Angeles are impacted.

China Telecom LAX-SHA CN2 GIA partial circuits failure.
( All CN2 LAX customers are impacted. )

TYO Pro:
- NCP S3 submarine fiber cut again. Impact CTGnet backbone.
- Higher latency (bypass HKG.)
- No ETR.

LAX Pro:
- CN2 backbone local fiber cut in LA Downtown.
- Packet loss and congestion happening.
- ETR 14:00 UTC.

Please note:

DMIT does not own those fiber and we do not have control on China Telecom’s backbone.

All the fiber cuts above only impact the China Telecom network.

DMIT backbone is normal so far.

We will keep watch on it and chase for update from China Telecom.

HKG Pro:
We are currently adjusting internet routing. Some ISPs have excessively long routing convergence times, leading to detours and blackouts. Thank you for your patience.

The time zone has been changed.

Some VM restarted outside of the scheduled maintenance window.
This was caused by a Kernel Panic.
Following the configuration corrections and kernel component updates, a final restart will still be required during the scheduled maintenance window.
We sincerely apologize for any inconvenience this may cause.

LAX.EB

AS9929:
We've already report fault case with China Unicom about AS9929 issue.
DMIT has enough bandwidth with China Unicom. The bandwidth expansion has been completed but DMIT will not disclosure peering capacity any more in the future. DMIT still maintains a 50% bandwidth reserve. The current packet loss and fluctuations in latency are tentatively attributed to issues with China Unicom's backbone network.

AS58807:
CMIN2 is heavily oversold (On China Mobile US-CN backbone end). It has reach the ceiling of their capacity. However, they still sell small amounts of bandwidth to small customers. DMIT still maintains a 20% bandwidth reserve.

Again:
AS9929 has fragile backbone, China Unicom is reconfiguring their network recently, this might also be a reason for the recent unstable. (nearly 10th each month.)
DMIT has report this issue since 2 days ago when it actually happened.
There is no useful information and no action has been done yet.
China Unicom has never change, they always like this.

AS58807:
CMIN2 is heavily oversell, they cannot delivery full of purchased burst bandwidth. They don't have upgrade plan in any place. To improve overall experience, DMIT will have to divert "some" CM traffic to CU 9929 to reduce the load for CMIN2 "after AS9929 back to normal".
Based on test, when the AS9929 is stable, the path of AS9808-AS4837-AS9929 will be stable and fast. There is no evidence shows cross-ISP rate limit.

AS9929 states, the packet loss and jitter caused by submarine fiber cut between Shanghai-Tokyo segment. They states (not us) it has been fixed now. (by detour traffic.)
Let's see how it goes.

To all customers:
DMIT resources are in a very critical status. That's the reason we pause all sales at this time to minimize the impact on existing customers.
We do apologize that there isn't any notice about the sales pause for weeks.

For all locations:
Our IPv4 address purchasing is under contract, we are getting more IPs very soon.
We are pushing it to move fast but we cannot guarantee when it will be ready.
Also once the IPv4 is transferred, we will publish it into the resource pool to release IP demand pressure and restore sales immediately.
This might bring some routing inconsistency (main for inbound.) and geolocation issues (short time advertisement, and IP database update delay).
We used to hold IP for more than 3-6 mo before it was put into the resource pool but this time we couldn't.

For LAX,
The hardware arrived last week, due to limited space, we are working on getting the new space and power. It takes a few more weeks.
Even after it's ready, we won't restore sales for large scale instances.
Because we need to scale up our storage cluster; the storage cluster is on the way to datacenter and it might take a little more times more once the computing node is ready.

For HKG,
Hardware is onsite, we are going to build it ASAP.
Once the IP resource is ready, the sales will be restored.

For TYO,
We are only waiting for more IPs.

LAX:
AS9929 IPv6 inbound is ready for LAX.Pro and LAX.EB

Update:
1. HKG.Pro plan (except discount plan) has been upgraded to HKG.AN5.Pro The product will be available to order very soon.
HKG AN5 is using DDR5 5600 9655P hardware.
1. Half of maintenance has been completed. The rest will be completed soon.

LAX:

From CMI NOC:
From 0:00 to 6:00 AM Hong Kong time on June 16th, NCP submarine cable will have a routine maintenance, which may affect IP transit service.