ATProto Browser
Experimental browser for the Atmosphere
Experimental browser for the Atmosphere
If you trained your tokenizer on Reddit data, but not your LLM then the problem could persist. It seems like the deletion step should be a solid win, except that the inference procedure changes a bit, so the current fast HuggingFace tokenizers can't be used.
Dec 20, 2024, 6:45 PM
{
"text": "If you trained your tokenizer on Reddit data, but not your LLM then the problem could persist. \n\nIt seems like the deletion step should be a solid win, except that the inference procedure changes a bit, so the current fast HuggingFace tokenizers can't be used.",
"$type": "app.bsky.feed.post",
"langs": [
"en"
],
"reply": {
"root": {
"cid": "bafyreictfbybzbhtvpo4cfus4h45zrh64b3qqgcnfr5phmn3wimmbrhgdm",
"uri": "at://did:plc:54lqvssae6v2kio2cu26yktz/app.bsky.feed.post/3ldr2b5ec722r"
},
"parent": {
"cid": "bafyreidys4w2xn4l5rxyrghph6upr72s7j62um64mz2ur5e2apm3sgeh6m",
"uri": "at://did:plc:54lqvssae6v2kio2cu26yktz/app.bsky.feed.post/3ldr2b5scgs2r"
}
},
"createdAt": "2024-12-20T18:45:49.351Z"
}