Skip to content

Use dailyEvents counter for all workspaces in limiter #584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
neSpecc merged 1 commit into from
Aug 26, 2026
Merged

Use dailyEvents counter for all workspaces in limiter #584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .env.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,5 @@ HAWK_CATCHER_TOKEN=
## If true, Grouper worker will send messages about new events to Notifier worker
IS_NOTIFIER_WORKER_ENABLED=false

## Comma-separated workspace ids that should use dailyEvents counters in Limiter quota checks. Use * for all workspaces.
LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS=

## Url for telegram notifications about workspace blocks and unblocks
TELEGRAM_LIMITER_CHAT_URL=
128 changes: 65 additions & 63 deletions workers/limiter/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,7 @@ export default class LimiterWorker extends Worker {

const since = Math.floor(new Date(workspace.lastChargeDate).getTime() / MS_IN_SEC);

const workspaceEventsCount = await this.getWorkspaceEventsCount(workspace, projects, since);
const workspaceEventsCount = await this.dbHelper.getEventsCountByProjectsUsingDailyEvents(projects, since);

this.logger.info(`workspace ${workspace._id} events count since last charge date: ${workspaceEventsCount}`);

Expand Down Expand Up @@ -328,68 +328,70 @@ export default class LimiterWorker extends Worker {
};
}

/**
* Returns workspace events count using the default raw counter or the
* dailyEvents-based counter when it is explicitly enabled for the workspace.
*
* For enabled workspaces both counters are computed and their results with
* timings are reported to Telegram to compare the algorithms during the
* testing period. The old counter is used as a fallback if the new one fails.
*
* @param workspace - workspace to count events for
* @param projects - workspace projects
* @param since - timestamp of the time from which we count the events
*/
private async getWorkspaceEventsCount(
workspace: WorkspaceWithTariffPlan,
projects: ProjectDBScheme[],
since: number
): Promise<number> {
if (!this.shouldUseDailyEventsCounter(workspace._id.toString())) {
return this.dbHelper.getEventsCountByProjects(projects, since);
}

const oldAlgoStartedAt = Date.now();
const oldAlgoCount = await this.dbHelper.getEventsCountByProjects(projects, since);
const oldAlgoTook = (Date.now() - oldAlgoStartedAt) / MS_IN_SEC;

try {
const newAlgoStartedAt = Date.now();
const newAlgoCount = await this.dbHelper.getEventsCountByProjectsUsingDailyEvents(projects, since);
const newAlgoTook = (Date.now() - newAlgoStartedAt) / MS_IN_SEC;

telegram.sendMessage(
`Workspace <b>${workspace.name}</b> event count:\n` +
`Old algo: ${oldAlgoCount}, took ${oldAlgoTook}sec\n` +
`New algo: ${newAlgoCount}, took ${newAlgoTook}sec`,
telegram.TelegramBotURLs.Limiter
);

return newAlgoCount;
} catch (error) {
HawkCatcher.send(error, {
workspaceId: workspace._id.toString(),
});

return oldAlgoCount;
}
}

/**
* Checks whether dailyEvents-based quota counting is enabled for the workspace
* via LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS environment variable —
* comma-separated workspace ids or `*` to enable it for every workspace.
*
* @param workspaceId - workspace id
*/
private shouldUseDailyEventsCounter(workspaceId: string): boolean {
const enabledWorkspaceIds = (process.env.LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS || '')
.split(',')
.map(id => id.trim())
.filter(Boolean);

return enabledWorkspaceIds.includes('*') || enabledWorkspaceIds.includes(workspaceId);
}
// Old raw counter with the opt-in switch, kept in case we need to roll back
//
// /**
// * Returns workspace events count using the default raw counter or the
// * dailyEvents-based counter when it is explicitly enabled for the workspace.
// *
// * For enabled workspaces both counters are computed and their results with
// * timings are reported to Telegram to compare the algorithms during the
// * testing period. The old counter is used as a fallback if the new one fails.
// *
// * @param workspace - workspace to count events for
// * @param projects - workspace projects
// * @param since - timestamp of the time from which we count the events
// */
// private async getWorkspaceEventsCount(
// workspace: WorkspaceWithTariffPlan,
// projects: ProjectDBScheme[],
// since: number
// ): Promise<number> {
// if (!this.shouldUseDailyEventsCounter(workspace._id.toString())) {
// return this.dbHelper.getEventsCountByProjects(projects, since);
// }
//
// const oldAlgoStartedAt = Date.now();
// const oldAlgoCount = await this.dbHelper.getEventsCountByProjects(projects, since);
// const oldAlgoTook = (Date.now() - oldAlgoStartedAt) / MS_IN_SEC;
//
// try {
// const newAlgoStartedAt = Date.now();
// const newAlgoCount = await this.dbHelper.getEventsCountByProjectsUsingDailyEvents(projects, since);
// const newAlgoTook = (Date.now() - newAlgoStartedAt) / MS_IN_SEC;
//
// telegram.sendMessage(
// `Workspace <b>${workspace.name}</b> event count:\n` +
// `Old algo: ${oldAlgoCount}, took ${oldAlgoTook}sec\n` +
// `New algo: ${newAlgoCount}, took ${newAlgoTook}sec`,
// telegram.TelegramBotURLs.Limiter
// );
//
// return newAlgoCount;
// } catch (error) {
// HawkCatcher.send(error, {
// workspaceId: workspace._id.toString(),
// });
//
// return oldAlgoCount;
// }
// }
//
// /**
// * Checks whether dailyEvents-based quota counting is enabled for the workspace
// * via LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS environment variable —
// * comma-separated workspace ids or `*` to enable it for every workspace.
// *
// * @param workspaceId - workspace id
// */
// private shouldUseDailyEventsCounter(workspaceId: string): boolean {
// const enabledWorkspaceIds = (process.env.LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS || '')
// .split(',')
// .map(id => id.trim())
// .filter(Boolean);
//
// return enabledWorkspaceIds.includes('*') || enabledWorkspaceIds.includes(workspaceId);
// }

/**
* Method that formats project list to html used in report messages
Expand Down
33 changes: 12 additions & 21 deletions workers/limiter/tests/index.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ describe('Limiter worker', () => {
expect(reportMessage).toContain(`${project1.name} (id: <code>${project1._id}</code>)`);
});

test('Should compute both counters and report the comparison to Telegram when dailyEvents counter is enabled', async () => {
test('Should count events via dailyEvents counters for every workspace', async () => {
/**
* Arrange
*/
Expand All @@ -349,45 +349,36 @@ describe('Limiter worker', () => {
});

/**
* Bucket for the day after the boundary day — counted only by the new algorithm
* Bucket for the day after the boundary day — counted via dailyEvents
*/
await db.collection(`dailyEvents:${project._id.toString()}`).insertOne({
groupHash: 'ade987831d0d0d167aeea685b49db164eb4e113fd027858eef7f69d049357f62',
groupingTimestamp: NEXT_MIDNIGHT_AFTER_LAST_CHARGE,
count: 7,
});

process.env.LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS = workspace._id.toString();

/**
* Act
*/
try {
const worker = new LimiterWorker();

await worker.start();
await worker.handle(REGULAR_WORKSPACES_CHECK_EVENT);
await worker.finish();
} finally {
delete process.env.LIMITER_DAILY_EVENTS_COUNTER_WORKSPACE_IDS;
}
const worker = new LimiterWorker();

await worker.start();
await worker.handle(REGULAR_WORKSPACES_CHECK_EVENT);
await worker.finish();

/**
* Assert — the new counter result is saved, both results are reported with timings
* Assert
*/
const workspaceInDatabase = await workspaceCollection.findOne({
_id: workspace._id,
});

expect(workspaceInDatabase.billingPeriodEventsCount).toBe(12); // 5 boundary-day events + 7 from dailyEvents

const comparisonMessage = (telegram.sendMessage as jest.Mock).mock.calls
.map(call => call[0])
.find(message => message.includes('Old algo'));

expect(comparisonMessage).toContain(`Workspace <b>${workspace.name}</b> event count:`);
expect(comparisonMessage).toMatch(/Old algo: 5, took [\d.]+sec/);
expect(comparisonMessage).toMatch(/New algo: 12, took [\d.]+sec/);
/**
* Counters comparison is not reported to Telegram anymore
*/
expect(telegram.sendMessage).not.toHaveBeenCalled();
});

test('Should not send a report when no projects are blocked or unblocked', async () => {
Expand Down
Loading