indeed_ai_job_search/Docs/Phase2/Indeed_Access_Spike.md

Indeed access spike notes

Phase: 2
Scope: Technical + policy constraints only — not authorization to ship scraping
Date: 2026-10-12

---

1. ToS / automation constraints (planning)

• Indeed’s terms generally prohibit unauthorized automated access / scraping of site content.
• App Store review and user trust require a permitted data story.
• A controlled spike that measures HTML fragility (CAPTCHA, rate limits, layout churn) may inform a don’t-build decision; it must not become a production path without permission.

Spike posture for this repo: no live scrape client. Fixtures simulate permitted JSON payloads.

---

2. Field coverage needed by the app

Mapped to JobListing:

Domain field	Required	Notes
indeedJobKey	Yes	Dedupe + save identity
title	Yes
company	Yes
location	Yes	Free-text OK
remoteType	Preferred	Map remote/hybrid/onsite/unknown
salary	Optional	min/max/currency/period
postedAt	Preferred	Relative date UI
snippet	Preferred	Detail / row preview
indeedURL	Yes	Must be Indeed TLD
availability	Preferred	open / unknown / unavailable
employment / experience	Optional	Filter support

---

3. Regional TLDs

Indeed hosts vary by country (examples): indeed.com, indeed.co.uk, pk.indeed.com, ca.indeed.com, etc.

Canonicalization rules (implemented in IndeedURLCanonicalizer):

• Accept only hosts matching indeed. / *.indeed. patterns
• Prefer HTTPS
• Extract job key from query jk= or known path segments when present
• Preserve regional host when opening in browser so the user lands on the correct locale

Country setting (Settings → Search defaults / criteria.country) should map to the vendor’s locale parameter and preferred TLD when the provider supports it.

---

4. Pagination, caching, rate limits (contract defaults)

Concern	Default (until vendor docs override)
Page size	25
Max pages per user search (UI)	4 (100 results) before “load more”
Cache TTL (listing metadata)	15 minutes in-memory (Phase 4+)
Saved-job revalidation TTL	24 hours (Phase 5)
Rate limit handling	Surface IndeedDataError.rateLimited(retryAfter:) — backoff UI
Empty page	Treat as end of results

---

5. Error taxonomy

Error	User-facing intent
unauthorized	Source credentials missing/invalid
forbidden	License / geo not allowed
rateLimited	Retry later
invalidResponse	Decode / schema failure
unsupportedFilter	Best-effort note (hide or label in UI later)
network	Offline / transport
cancelled	Task cancelled
nonIndeedURL	Mapper rejected non-Indeed link

---

6. indeedJobKey extraction

Priority:

1. Explicit jobKey / jk field from provider DTO
   
2. Query item jk on Indeed URL
   
3. Stable path token when vendor uses /viewjob?jk=
   
4. Fallback: hash of canonical Indeed URL (last resort; prefer real key)

---

7. Technical viability checklist (when a vendor is chosen)

Measure against sandbox credentials (still no scrape):

• Latency p50 / p95 for page of 25
• Geo coverage for target countries (e.g. Pakistan, US, UK)
• Remote/hybrid signals accuracy
• Salary presence rate
• Pagination cursor vs offset
• Expired listing signals

Record results in this doc when a vendor sandbox is available.

---

8. Outcome for Phase 2

• DADR recommends licensed / partner Indeed-only channel
  
• In-repo: protocol + mapper + fixtures + unit tests
  
• No production network adapter until DADR approval + contract