| QID | Question | P@k | R@k | RR | nDCG |
|---|---|---|---|---|---|
| q01 | How much does the Growth plan cost per month? | 0.20 | 1.00 | 0.50 | 0.63 |
| q02 | What encryption is used for data at rest? | 0.20 | 1.00 | 1.00 | 1.00 |
| q03 | How long are audit logs kept? | 0.20 | 1.00 | 1.00 | 1.00 |
| q04 | How many days of PTO can staff take each year? | 0.20 | 1.00 | 1.00 | 1.00 |
| q05 | What is the API rate limit on the Growth plan? | 0.20 | 1.00 | 1.00 | 1.00 |
| q06 | Which identity providers are supported for single sign-on, and which plan includes SCIM provisioning? | 0.40 | 1.00 | 1.00 | 1.00 |
| q07 | Can I get my money back if I cancel an annual subscription? | 0.00 | 0.00 | 0.00 | 0.00 |
| q08 | How quickly does support respond to a critical outage on the Enterprise plan? | 0.00 | 0.00 | 0.00 | 0.00 |
| q09 | When is a root cause analysis published after an incident? | 0.20 | 1.00 | 1.00 | 1.00 |
| q10 | Does Northwind offer a discount for paying yearly? | 0.20 | 1.00 | 1.00 | 1.00 |
| QID | Answer & verified claims | Faith. | Halluc. | Relev. |
|---|---|---|---|---|
| q01 | The Growth plan costs **$199 per month**.
If you choose annual billing, you'll receive a 20% discount, bringing the annual cost to **$1,592.80** (or approximately **$132.73 per month**). • The Growth plan costs $199 per month. — supported • Annual billing provides a 20% discount. — supported • With annual billing, the Growth plan's annual cost is $1,592.80. — contradicted • With annual billing, the Growth plan costs approximately $132.73 per month. — contradicted | 0.50 | 0.50 | 1.00 |
| q02 | Data at rest is encrypted with **AES-256**. • Data at rest is encrypted with AES-256. — supported | 1.00 | 0.00 | 1.00 |
| q03 | Audit logs are retained for **90 days** on all plans. If you need longer retention, log export to your own storage is available on Enterprise plans. • Audit logs are retained for 90 days. — supported • The 90-day audit log retention applies to all plans. — supported • Log export to the customer's own storage is available on Enterprise plans. — supported | 1.00 | 0.00 | 1.00 |
| q04 | Based on the company policy, full-time employees receive **35 paid days of PTO per year**, which includes:
- **25 days** of annual leave
- **10 days** of sick leave
This is in addition to public holidays observed in their country of employment. • Full-time employees receive 35 paid days of PTO per year. — unsupported • Employees receive 25 days of annual leave. — supported • Employees receive 10 days of sick leave per year. — supported • Leave is in addition to public holidays observed in the employee's country of employment. — supported | 0.75 | 0.25 | 0.75 |
| q05 | The API rate limit on the Growth plan is **600 requests per minute**, measured per workspace token. • The API rate limit on the Growth plan is 600 requests per minute. — supported • The rate limit is measured per workspace token. — supported | 1.00 | 0.00 | 1.00 |
| q06 | Based on the documentation:
**Supported Identity Providers for SSO:**
- Okta
- Microsoft Entra ID (Azure AD)
- Google Workspace
Northwind Analytics supports SAML 2.0 single sign-on with these providers.
**SCIM Provisioning:**
SCIM user provisioning and deprovisioning is available on the **Enterprise plan only**. • Okta is a supported identity provider for SSO. — supported • Microsoft Entra ID (Azure AD) is a supported identity provider for SSO. — supported • Google Workspace is a supported identity provider for SSO. — supported • Northwind Analytics supports SAML 2.0 single sign-on with these providers. — supported • SCIM user provisioning and deprovisioning is available on the Enterprise plan only. — supported | 1.00 | 0.00 | 1.00 |
| q07 | Based on our refund policy, new subscriptions have a 14-day money-back guarantee on the first purchase, no questions asked. This applies regardless of whether you choose monthly or annual billing.
However, I'd recommend contacting our support team if you're considering cancellation, as they can discuss your specific situation and ensure you're getting the most value from your subscription. • New subscriptions have a 14-day money-back guarantee on the first purchase, no questions asked. — supported • The 14-day money-back guarantee applies regardless of whether the customer chooses monthly or annual billing. — unsupported • There is a support team that can be contacted to discuss a customer's specific situation regarding cancellation. — unsupported | 0.33 | 0.67 | 0.50 |
| q08 | Based on the provided context, I can confirm that **support is available 24/7 for Enterprise workspaces**, which means your critical outage will be addressed around the clock.
However, the context doesn't specify the actual response time or SLA for critical issues on the Enterprise plan. To get the exact response time commitment for critical outages, you would need to check your specific Enterprise support SLA agreement or contact our support team directly. • Support is available 24/7 for Enterprise workspaces. — supported • A critical outage on the Enterprise plan will be addressed around the clock. — unsupported • The context does not specify the actual response time or SLA for critical issues on the Enterprise plan. — supported • An exact response time commitment for critical outages can be obtained from the customer's specific Enterprise support SLA agreement or by contacting the support team. — unsupported | 0.50 | 0.50 | 1.00 |
| q09 | According to the incident response documentation, a root cause analysis (RCA) is published within **5 business days after the incident is resolved**, and it includes corrective actions and owners. • A root cause analysis (RCA) is published within 5 business days after the incident is resolved. — supported • The RCA includes corrective actions and owners. — supported • This information comes from the incident response documentation. — supported | 1.00 | 0.00 | 1.00 |
| q10 | Yes, Northwind Analytics offers a 20% discount on the Starter and Growth plans when you choose annual billing. The discount is applied automatically at checkout when you select annual billing. • Northwind Analytics offers a 20% discount on the Starter and Growth plans when annual billing is chosen. — supported • The discount is applied automatically at checkout when annual billing is selected. — supported | 1.00 | 0.00 | 1.00 |