20# MFS — find / read across configured sources
21
22## 1. What MFS is
23
24A retrieval layer that exposes many kinds of content as a unified path
25tree and makes that tree searchable through one hybrid index:
26
27- **One CLI (mfs), one mental model.** Local dir, Postgres, GitHub repo,
28 Slack workspace, S3 bucket, BigQuery dataset — all addressed as paths
29 under their <scheme>:// URI. Same verbs everywhere: `ls / tree / cat /
30 head / tail / grep / search / export`.
31- **One hybrid index.** Dense vectors (semantic) + BM25 (keyword) fused
32 per query — covers conceptual recall and exact-token recall in one call.
33- **POSIX-style locators.** Every search hit carries a locator that
34 reopens the exact unit: {"lines":[s,e]} for text/code, a PK dict for
35 rows/issues/threads.
36
37## 2. When to use MFS — and when NOT to
38
39| Situation | Use MFS? |
40|---|---|
41| 1000+ files / rows / pages, you don't know where the answer is | ✅ |
42| Cross-source question ("any past tickets / commits / RFCs about X") | ✅ --all |
43| Concept-style query that won't match literally | ✅ --mode semantic |
44| You already know the exact file + roughly where to look | ❌ plain cat/grep |
45| Exact identifier / error code in 5 files you can list | ❌ plain grep/rg |
46| Real-time tailing of a live log | ❌ index lags ingest |
47| The source isn't in MFS yet | wrong skill, use mfs-ingest to register first |
48
49**Rule:** use the smallest tool that answers the question. MFS pays off
50when the scope is too big for rg.
51
52**Borderline — ASK the user:**
53
54| Ask | Likely answer | Why |
55|---|---|---|
56| "Summarise these 10 PDFs" | ✅ mfs search + cat --peek per hit | each PDF gets a converted_md artifact + searchable chunks |
57| "Find similar tickets to this one" | ✅ paste the ticket text as the search query | semantic over row_text does similarity matching |
58| "Watch for new slack messages" | ❌ no watch capability; use Slack's API | index lags ingest |
59| "Look up user 12345" | ❌ mfs cat <source> --locator '{"id":12345}' directly (skip search) | one-record-by-id doesn't need ranking |
60
61## 3. Pre-flight — confirm the source is indexed
62
63Before running any query, especially on cross-source asks:
64
65```bash
66mfs status # server up? any connectors registered?
67mfs connector inspect <uri> # this connector's object/job summary
68mfs ls <uri> --json # per-entry capabilities + indexable / search_status
69```
70
71- Server unreachable → tell user to start it (mfs serve start if
72 self-hosted), or this skill can't proceed.
73- connectors empty → user hasn't ingested anything yet. **Redirect to
74 mfs-ingest** — don't try to search nothing.
75- search_status: unavailable for the target URI → only grep / ls /
76 cat work; offer those or redirect to mfs-ingest for a re-sync.
77- building → sync in flight; fall back to mfs grep (works without an
78 index) until done.
79- partial → recall incomplete but usable; flag the caveat to the user.
80
81## 4. The core workflow: search → locate → browse
82
83```
84 search locate browse
85 ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────────┐
86 │ semantic + BM25 │ → │ result has lines │ → │ cat --range / cat │
87 │ finds candidates│ │ or a locator │ │ --peek to confirm │
88 └──────────────────┘ └──────────────────┘ └─────────────────────┘
89```
90
91On large corpora this loop is the whole point: read only the part that
92matters. On small corpora it's still fine, just lighter.
93
94Concrete:
95
961. **Search:**
97 ```bash
98 mfs search "<what the user actually wants>" <path-or-uri> --top-k 10
99 ```
1002. **Locate** — every hit's envelope carries locator:
101 - text/code → {"lines":[start,end]} → mfs cat <source> --range start:end
102 - structured (row/issue/thread) → PK dict → mfs cat <source> --locator '{...}'
103 - once-per-object (dir/schema summary, image VLM) → null → mfs cat <source>
1043. **Browse** — verify only what's needed:
105 ```bash
106 mfs cat --peek <file> # outline (headings / function signatures)
107 mfs cat --skim <file> # peek + one-line summaries per section
108 mfs head -n 20 <uri> # first records of a structured object
109 mfs tree <uri> -L 2 # subtree shape
110 ```
111
112## 5. Index requirement rules of thumb
113
114- mfs search **requires** an index.
115- mfs grep works **without** — pushdown → BM25 → linear scan fallback.
116- mfs ls / tree / cat / head / tail browse **without** an index.
117
118## 6. Search modes
119
120mfs search defaults to hybrid. Override only when you know why.
121
122| --mode | Mechanic | When |
123|---|---|---|
124| **hybrid** *(default)* | dense + BM25 fused with RRF | almost always |
125| semantic | dense only | conceptual query, wording won't match literally |
126| keyword | BM25 only | exact-term (config key, error code) without semantic drift |
127
128Other useful flags:
129
130- --top-k N — default 10; raise to 20-30 on a weak first round.
131- --all — search every registered connector. Otherwise scope to a path/URI prefix.
132- --kind <list> — restrict chunk kinds (row_text, thread_aggregate,
133 body, summary, vlm_description, …).
134- --collapse — keep only the top-scoring chunk per object; later chunks
135 from the same source are dropped, not merged. Recall stays as-is (the
136 query still hits the same candidates), but the visible result count can
137 fall below --top-k — collapse is a post-filter, not a re-rank. If you
138 want N distinct objects, raise --top-k (e.g. --top-k 30 --collapse).
139
140### --all: when yes, when no
141
142- ✅ Cross-source recall — "any past tickets / commits / RFCs / slack about X".
143- ❌ You know the source — scope to slack://; postgres + jira + docs together aren't comparable.
144- ⚠ More than ~5 registered connectors — ASK the user whether to fan out
145 widely or scope to the 2-3 likeliest sources first.
146
147## 7. Decision tree — pick the smallest useful tool
148
149| Signal in the ask | Sub-task | Use |
150|---|---|---|
151| natural-language question / sentence | exploratory | mfs search "<q>" <scope> |
152| paraphrased / conceptual wording | semantic-only | mfs search --mode semantic |
153| exact identifier / config key / unique phrase | literal anchor | mfs grep "<lit>" <path> (or rg) |
154| filename / directory pattern | path lookup | find / shell glob / fd |
155| known file + needs outline | structural overview | mfs cat --peek <file> |
156| known file + compact summary | dense overview | mfs cat --skim <file> |
157| search hit + surrounding context | reopen | mfs cat <file> --range s:e |
158| structured hit (row/issue/thread) | reopen by PK | mfs cat <source> --locator '{...}' |
159| several close candidates | compare | mfs cat --peek each, then pick |
160| single record + known key | no-search lookup | mfs cat <source> --locator '{"id":12}' |
161| first / last N | sample | mfs head -n N / mfs tail -n N |
162| subtree shape | orient | mfs tree -L 2 <uri> |
163| full object for offline tooling | export | mfs export <uri> <file> |
164
165mfs search requires an explicit scope or --all.
166
167## 8. Command cheat sheet
168
169### Search
170
171```bash
172mfs search "<query>" <path-or-uri> # default: hybrid, top-k=10
173mfs search "<query>" --all # whole namespace
174mfs search "<query>" <path> --top-k 20 # more candidates
175mfs search "<query>" <path> --mode semantic # dense-only
176mfs search "<query>" <path> --mode keyword # BM25-only
177mfs search "<query>" <path> --kind row_text # restrict chunk kinds
178mfs search "<query>" <path> --collapse # keep top hit per object (post-filter, may return < top-k)
179```
180
181### Grep
182
183```bash
184mfs grep "<pattern>" <path> # pushdown -> BM25 -> linear
185```
186
187**mfs grep is not grep.** The three-tier dispatch is:
188
1891. **Pushdown** — for structured connectors (postgres, mongo,
190 jira, …) the pattern is shipped to the source as a LIKE / regex
191 filter. Literal-exact, token-level; no regex on most structured
192 connectors.
1932. **BM25** over indexed objects — for body/code/document chunks
194 already in Milvus, the pattern is fed through the same sparse
195 index search --mode keyword uses. That is a tokenized,
196 ranked lookup, **not** a literal substring scan: an analyzer split
197 like getUserId → get, user, id will rank
198 userId as a hit; a CJK pattern with no analyzer match returns
199 nothing even when the literal bytes are present. If you need
200 "does this exact byte string appear anywhere?", mfs export the
201 object and run rg locally — mfs grep has no "force linear over
202 indexed objects" flag.
2033. **Linear scan** — only for not-indexed files in scope (file
204 connector before mfs add). True substring / regex.
205
206For exact-exhaustive on a huge structured object, mfs export then
207local grep / rg.
208
209### Read
210
211```bash
212mfs cat <path> # full content (refused if "lazy")
213mfs cat <path> --range A:B # lines A..B-1 (1-based, end-exclusive)
214mfs cat <path> --locator '{"id":12}' # reopen a structured record
215mfs cat <path> --peek # outline only
216mfs cat <path> --skim # peek + per-section summaries
217mfs cat <path> --meta # stat-style, not content
218```
219
220Density ladder:
221
222| Mode | Use it when |
223|---|---|
224| --peek | "show me the outline" |
225| --skim | + one-line summary per section, still concise |
226| (default) | full content; small file or really need it |
227| --range A:B | already know which lines matter (e.g. search hit) |
228
229```bash
230mfs head -n 50 <path> # first 50 lines/records
231mfs tail -n 50 <path> # last 50; native-accel reverse read
232```
233
234For a lazy rows.jsonl / messages.jsonl, head is how to see record
235shape without paying full-scan cost.
236
237### Browse
238
239```bash
240mfs ls <uri> # one level
241mfs tree <uri> -L 2 # depth-bounded recursive
242```
243
244NOT a substitute for search when the target is unknown and conceptual.
245
246### Export
247
248```bash
249mfs export <uri> <out-file> # full object to disk for jq/awk pipelines
250```
251
252cat of a huge lazy object is refused — use export for bulk processing.
253
254### Status (useful before AND during search work)
255
256```bash
257mfs status # server + all connectors
258mfs connector inspect <uri> # one connector's object/job summary
259mfs connector list # list registered connectors
260mfs job list # recent indexing jobs (background re-syncs)
261```
262
263Always prefer --json when output will be parsed.
264
265## 9. Weak results → recover, don't thrash
266
267If top hits look off-topic:
268
2691. **Rewrite** with synonyms / domain terms. ASK the user for the domain
270 term they'd actually use if vague. One clarifier beats five blind queries.
2712. **Raise --top-k** to compare distinct candidates.
2723. **mfs cat --peek** the top few to compare structure.
2734. **Switch mode** — semantic if hybrid was keyword-noisy; keyword if
274 specific terms should be the anchor.
2755. **Then** literal grep — only if the task has a real literal anchor
276 (error code, config key, identifier).
277
278Literal search is a *different* tool, not a stronger version of semantic.
279
280## 10. Candidate selection
281
282Think at object level, not just chunk level:
283
284- **Merge** repeated hits from the same object into one candidate.
285- **Compare** the top distinct candidates' --peek when titles or
286 snippets look adjacent.
287- **Prefer** an object whose main topic directly matches the request
288 over a broad overview that mentions it.
289- **Multi-part prompts** (two entities, setup + troubleshooting, migration
290 source + target) — check whether more than one object is needed.
291
292```bash
293mfs search "<query>" <path> --top-k 20
294mfs cat --peek <candidate-a>
295mfs cat --peek <candidate-b>
296mfs cat <best> --range <start>:<end>
297```
298
299## 11. Anti-patterns
300
301- **Don't grep to "confirm" a successful semantic hit.** The hit's
302 snippet IS the source content; trust it.
303- **Don't read a whole large file** when --peek / --skim / --range
304 can answer.
305- **Don't blindly pick rank #1** when #1-#3 are clearly different objects.
306- **Don't stop at one match** if the prompt mentions multiple entities.
307- **Don't search the same vague words after a weak first round** — fix
308 the query or escalate to literal anchors.
309- **Don't cat a lazy object** (DB rows.jsonl, SaaS records.jsonl,
310 chat messages.jsonl). Use head, --range, --locator, or export.
311- **Don't use MFS for sources you'd just clone/download anyway** — pull
312 locally and use the file connector.
313
314## 12. When search returns nothing on a freshly indexed connector
315
316This is the most common diagnostic case. Walk this ladder, stop on first
317hit:
318
319```bash
320# 1. THIS connector's object/job summary
321mfs connector inspect <uri>
322
323# 2. failed sync jobs?
324mfs job list
325mfs job show <job-id> # if any failed, read the job's error field
326
327# 3. per-object granularity
328mfs ls <uri> --json # each entry's search_status
329
330# 4. JSON error code path
331# If --json output carried a code field, see reference/error-codes.md
332
333# 5. otherwise treat as query-level — §9 above
334```
335
336| Signal | Meaning | Action |
337|---|---|---|
338| building | sync in flight | wait, or mfs grep until done |
339| partial | chunks dropped (chunk_max / max_read_rows) | usable but incomplete; user may want to re-ingest with raised caps (→ redirect to mfs-ingest) |
340| unavailable | nothing indexed | only grep / ls / cat work; redirect to mfs-ingest |
341| available but smoke search empty | wrong text_fields / source empty / wrong scope | check reference/connectors/<scheme>.md for that connector's shape; mfs cat a known object to confirm content |
342
343When the diagnosis points to "ingest config is wrong" or "needs re-sync
344with different settings" — **don't try to fix it from this skill**. Tell
345the user to invoke mfs-ingest for that connector.
346
347## 13. Reference routing
348
349These reference files are loaded ONLY when the situation matches — don't
350open speculatively.
351
352- **[reference/json-envelope.md](reference/json-envelope.md)** —
353 WHEN parsing a --json search/grep result and the locator shape is
354 unfamiliar (composite PKs, thread_ts, nested keys); OR when uncertain
355 how to feed a hit back into mfs cat (range vs locator dispatch).
356
357- **[reference/error-codes.md](reference/error-codes.md)** —
358 WHEN an mfs command returned --json error output with a code
359 field. Read the message first — don't open just because a command
360 failed.
361
362- **reference/connectors/<scheme>.md** —
363 WHEN searching a specific connector and you need its tree shape, record
364 field semantics, locator format, or search-strategy tips. STOP and read
365 the matching one BEFORE guessing how that source enumerates objects
366 or what fields its records carry. Schemes: file, web, s3,
367 gdrive, postgres, mysql, snowflake, bigquery, mongo,
368 github, jira, linear, hubspot, notion,
369 zendesk, slack, discord, gmail, feishu.
370
371Runtime capability for a specific URI is queried structurally via
372mfs ls <uri> --json; the static per-connector references describe what
373the connector exposes by design.
374