Observed IO error from Ceph in Hong Kong location.
It has been resolved.

HKG Maintenance

Start time: UTC+8 2024/04/19 02:00
Scope of Impact: All HKG hosts
Purpose: Upgrade NICs to CX5 Dual Port 100G, resolve high load issues on specific nodes, and upgrade server firmware.
Estimated Completion Time: Approx. 30 minutes per host

All completed. The VMs are booting in sequence.

DIRECT MESSAGE FROM RETN.NET [HKG]

Failure Notification

Description:
Our regional capacity is significantly reduced due to submarine fiber cuts in the Asia region.
During peak hours, multiple destinations may experience overutilization, resulting in packet loss and increased RTT.

Currently, we have registered the following emergency cases:
- Subsea fiber cut on Hong Kong - Taipei section. Started: 15.02 | ETR: 30.04
- Subsea fiber cut on Milan - Singapore section. Started: 24.02 | ETR: 19.04
- Subsea fiber cut on Hong Kong - Singapore section. Started: 10.03 | ETR: 13.05.
- Subsea fiber cut on Hong Kong - Taipei section. Started: 13.03 | approximate ETR: April END.
– Subsea fiber cut on Hong Kong - Singapore section. Started: 08.03 | ETR: 18.05.
- Subsea fiber cut on Hong Kong - Seoul section. Started: 03.04 | ETR: Unknown.

- Emergency situation on the Marseille - Singapore section. Start: 19.04 | Expected recovery: Unknown
- Emergency situation on the Sofia - Singapore section. Start: 19.04 | Expected recovery: Unknown

China Unicom (the Claimant) was unsuccessful in substantiating their claims in the arbitration case against DMIT (the Respondent) for the provision of services AS4837 and AS10099, valued at half a million US Dollars. Ultimately, it was determined that CUA should reimburse DMIT $x2,880.xx(5 digits) USD in the practical terms.

Initially, DMIT had entered into a significant contract with CUA for close to half a million dollars with the aim of joint development and exploring new market opportunities.

However, our plans changed due to unforeseen issues. CUA failed to fulfill the agreed-upon services and unilaterally altered the order, leading to the failed of DMIT's operations in San Jose and degrading our service quality in Tokyo.

CUA denied all liability and sought an arbitration ruling to compel DMIT to reimburse CUA for the full contract value.

As a leading provider of optimized routing solutions from the US to China, DMIT is disheartened by these events and the unreasonable business practices adopted by CUA. We regret to inform that we probably will no longer be working with CUA moving forward.

DMIT wants to share this story to all our partners; stay alert.

DMIT observed CN2 end user routes is now bypassing AS23764 since 3 days ago.

AS4809 is customer of AS1299 in US and EU; but AS23764 is not. This lead to the latency increase recently. DMIT will try our best to fix this ASAP.

Standard.inet.0: 946707 destinations, 2465729 routes (946093 active, 0 holddown, 467099 hidden)
+ = Active Route, - = Last Active, * = Both

58.32.0.0/17 *[BGP/170] 11:39:07, MED 154110, localpref 100
AS path: 174 23764 4809 4812 I, validation-state: unknown
> to _ via _
[BGP/170] 3d 22:49:21, MED 0, localpref 100
AS path: 1299 3356 23764 4809 4812 I, validation-state: unknown
> to _ via _

Temporarily detour CTGnet CN2 in Hong Kong for Premium customer, because of Subnet DDoS attack.

There are 0.5~1Mbps NTP to 4 of our subnets in Hong Kong which is about 700M~1Gbps inbound attack which is almost congested our network and the peer between AS10099 and AS23764.

Already ask CTGnet to check status of their Anti-DDoS system.

Resolved.

Observed HKG EB network issue; working on it. It takes time to work with ISP to fix that out.

DMIT is working with ISP on solving the problem. It might takes a long while.

Observed AS174 outage on all locations.

All the HKG EB customers who purchased with current price (puchased after change Lite to EB) could have reimbursement after problem be solved.

Temporarily detour EB to other network profile.

also observed AS1299 packet loss. There are major network issues happening. We have to wait at least one of ISPs to solve their issue.

Reimbursement incl all possible solution not just refund. Incl offer credit, discount, return value based on time.

ETR: within June.

Scheduled Maintenance

A)
Impact: HKG Pro
Time: June 20 3AM ~ 10AM, 2024 Hong Kong Time.
Durition: upto 1 hrs

B)
Impact: HKG all
Time: June 20 6AM ~ 10AM, 2024 Hong Kong Time.
Durition: upto 30min

- Optimize configuration to make it adapter DMIT's new architecture.
- Enhance design of MPLS Backbone.

You might experience network downgrading, short time over all black out, or partial destination unreachable.