| QID | Question | P@k | R@k | RR | nDCG |
|---|---|---|---|---|---|
| q01 | How much does the Growth plan cost per month? | 0.20 | 1.00 | 0.50 | 0.63 |
| q02 | What encryption is used for data at rest? | 0.20 | 1.00 | 1.00 | 1.00 |
| q03 | How long are audit logs kept? | 0.20 | 1.00 | 1.00 | 1.00 |
| q04 | How many days of PTO can staff take each year? | 0.20 | 1.00 | 0.50 | 0.63 |
| q05 | What is the API rate limit on the Growth plan? | 0.20 | 1.00 | 1.00 | 1.00 |
| q06 | Which identity providers are supported for single sign-on, and which plan includes SCIM provisioning? | 0.40 | 1.00 | 1.00 | 1.00 |
| q07 | Can I get my money back if I cancel an annual subscription? | 0.20 | 1.00 | 1.00 | 1.00 |
| q08 | How quickly does support respond to a critical outage on the Enterprise plan? | 0.20 | 1.00 | 0.50 | 0.63 |
| q09 | When is a root cause analysis published after an incident? | 0.20 | 1.00 | 1.00 | 1.00 |
| q10 | Does Northwind offer a discount for paying yearly? | 0.20 | 1.00 | 1.00 | 1.00 |
| QID | Answer & verified claims | Faith. | Halluc. | Relev. |
|---|---|---|---|---|
| q01 | The Growth plan costs **$199 per month**. • The Growth plan costs $199 per month. — supported | 1.00 | 0.00 | 1.00 |
| q02 | According to the security documentation, data at rest is encrypted with **AES-256**. • Data at rest is encrypted with AES-256. — supported • This information comes from the security documentation. — supported | 1.00 | 0.00 | 1.00 |
| q03 | Audit logs are retained for **90 days** on all plans. • Audit logs are retained for 90 days. — supported • The 90-day audit log retention applies to all plans. — supported | 1.00 | 0.00 | 1.00 |
| q04 | Full-time employees can take **35 paid days of PTO per year**, which consists of:
- **25 days** of annual leave
- **10 days** of sick leave
This is in addition to public holidays observed in their country of employment. • Full-time employees can take 35 paid days of PTO per year — unsupported • Full-time employees receive 25 paid days of annual leave per year — supported • Employees receive 10 paid days of sick leave per year — supported • This leave is in addition to public holidays observed in the employee's country of employment — supported | 0.75 | 0.25 | 1.00 |
| q05 | The API rate limit on the Growth plan is **600 requests per minute**, measured per workspace token. • The API rate limit on the Growth plan is 600 requests per minute. — supported • The rate limit is measured per workspace token. — supported | 1.00 | 0.00 | 1.00 |
| q06 | Based on the documentation:
**Supported Identity Providers for SSO:**
- Okta
- Microsoft Entra ID (Azure AD)
- Google Workspace
Northwind Analytics supports SAML 2.0 single sign-on with these providers.
**SCIM Provisioning:**
SCIM user provisioning and deprovisioning is available on the **Enterprise plan only**. • Okta is a supported identity provider for SSO — supported • Microsoft Entra ID (Azure AD) is a supported identity provider for SSO — supported • Google Workspace is a supported identity provider for SSO — supported • Northwind Analytics supports SAML 2.0 single sign-on with these providers — supported • SCIM user provisioning and deprovisioning is available on the Enterprise plan only — supported | 1.00 | 0.00 | 1.00 |
| q07 | Yes, you can get a refund if you cancel an annual subscription within 30 days of a renewal. In that case, you'll receive a prorated refund for the unused period.
Refund requests should be sent to billing@northwind.example and are processed within 5 to 10 business days to your original payment method. • You can get a refund if you cancel an annual subscription within 30 days of a renewal. — supported • The refund is a prorated refund for the unused period. — supported • Refund requests should be sent to billing@northwind.example. — supported • Refunds are processed within 5 to 10 business days. — supported • Refunds are issued to the original payment method. — supported | 1.00 | 0.00 | 1.00 |
| q08 | For a critical outage (Sev-1 incident) on the Enterprise plan, support targets a **first response within 1 hour**.
Additionally, during a Sev-1 incident, the public status page is updated at least every 30 minutes until resolution, keeping you informed of progress. • On the Enterprise plan, a critical outage is classified as a Sev-1 incident (full outage). — supported • For a Sev-1 incident on the Enterprise plan, support targets a first response within 1 hour. — supported • During a Sev-1 incident, the public status page is updated at least every 30 minutes until resolution. — supported | 1.00 | 0.00 | 1.00 |
| q09 | According to the incident response policy, a root cause analysis (RCA) is published within **5 business days after the incident is resolved**, and it includes corrective actions and owners. • A root cause analysis (RCA) is published within 5 business days after the incident is resolved. — supported • The RCA includes corrective actions and owners. — supported • This is stated in the incident response policy. — supported | 1.00 | 0.00 | 1.00 |
| q10 | Yes, Northwind offers a 20% discount on the Starter and Growth plans when you choose annual billing. The discount is applied automatically at checkout when you select annual billing. • Northwind offers a 20% discount on the Starter and Growth plans with annual billing. — supported • The discount is applied automatically at checkout when annual billing is selected. — supported | 1.00 | 0.00 | 1.00 |